[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463317683 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003306] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008818] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014068] *** VALIDATE selinux *** [ 0.022643] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027294] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031104] *** VALIDATE tmpfs *** [ 0.032454] *** VALIDATE proc *** [ 0.033257] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.037215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039015] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045435] debug: unmapping init [mem 0xffffffff90c59000-0xffffffff90c60fff] [ 0.047826] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056292] rcu: Hierarchical SRCU implementation. [ 0.058416] smp: Bringing up secondary CPUs ... [ 0.059624] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063334] smp: Brought up 1 node, 4 CPUs [ 0.065025] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141235] node 0 deferred pages initialised in 71ms [ 0.145096] devtmpfs: initialized [ 0.147274] x86/mm: Memory block size: 128MB [ 0.149762] gcov: version magic: 0x41383552 [ 0.152316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157278] pinctrl core: initialized pinctrl subsystem [ 0.158000] [ 0.158000] ************************************************************* [ 0.160014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162012] ** ** [ 0.163014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.169010] ** IOMMU data structures, which may compromise security on ** [ 0.171015] ** your system. ** [ 0.174013] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.179012] ** kernel, report this immediately to your vendor! ** [ 0.181012] ** ** [ 0.184014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186009] ************************************************************* [ 0.188635] NET: Registered protocol family 16 [ 0.191416] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200103] cpuidle: using governor menu [ 0.201527] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204735] PCI: Using configuration type 1 for base access [ 0.206279] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223074] cryptd: max_cpu_qlen set to 1000 [ 0.225287] ACPI: Added _OSI(Module Device) [ 0.228014] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243581] ACPI: Interpreter enabled [ 0.244056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245014] ACPI: Using IOAPIC for interrupt routing [ 0.247118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250289] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258822] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272389] acpiphp: Slot [2] registered [ 0.273105] acpiphp: Slot [5] registered [ 0.275114] acpiphp: Slot [6] registered [ 0.276214] acpiphp: Slot [7] registered [ 0.278110] acpiphp: Slot [8] registered [ 0.279118] acpiphp: Slot [9] registered [ 0.281115] acpiphp: Slot [10] registered [ 0.282104] acpiphp: Slot [3] registered [ 0.283095] acpiphp: Slot [4] registered [ 0.285093] acpiphp: Slot [11] registered [ 0.286102] acpiphp: Slot [12] registered [ 0.288117] acpiphp: Slot [13] registered [ 0.289091] acpiphp: Slot [14] registered [ 0.290145] acpiphp: Slot [15] registered [ 0.292110] acpiphp: Slot [16] registered [ 0.293087] acpiphp: Slot [17] registered [ 0.294165] acpiphp: Slot [18] registered [ 0.296089] acpiphp: Slot [19] registered [ 0.297101] acpiphp: Slot [20] registered [ 0.299078] acpiphp: Slot [21] registered [ 0.301092] acpiphp: Slot [22] registered [ 0.303101] acpiphp: Slot [23] registered [ 0.305179] acpiphp: Slot [24] registered [ 0.307117] acpiphp: Slot [25] registered [ 0.309105] acpiphp: Slot [26] registered [ 0.311104] acpiphp: Slot [27] registered [ 0.313113] acpiphp: Slot [28] registered [ 0.314076] acpiphp: Slot [29] registered [ 0.316072] acpiphp: Slot [30] registered [ 0.317093] acpiphp: Slot [31] registered [ 0.318081] PCI host bridge to bus 0000:00 [ 0.319015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336325] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341112] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344366] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373815] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379931] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.401025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415086] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458400] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494339] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.513014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.537642] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.552014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.559016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590501] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.602015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641589] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.675000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.684134] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687432] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690435] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702218] iommu: Default domain type: Passthrough [ 0.703461] SCSI subsystem initialized [ 0.705156] ACPI: bus type USB registered [ 0.706124] usbcore: registered new interface driver usbfs [ 0.708080] usbcore: registered new interface driver hub [ 0.710075] usbcore: registered new device driver usb [ 0.712257] pps_core: LinuxPPS API ver. 1 registered [ 0.714012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718056] PTP clock support registered [ 0.721058] EDAC MC: Ver: 3.0.0 [ 0.723156] PCI: Using ACPI for IRQ routing [ 0.724510] NetLabel: Initializing [ 0.726013] NetLabel: domain hash size = 128 [ 0.727009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730077] NetLabel: unlabeled traffic allowed by default [ 0.733088] vgaarb: loaded [ 0.734238] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744433] clocksource: Switched to clocksource kvm-clock [ 0.853194] VFS: Disk quotas dquot_6.6.0 [ 0.855195] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.858308] *** VALIDATE ramfs *** [ 0.859885] *** VALIDATE hugetlbfs *** [ 0.861798] pnp: PnP ACPI init [ 0.865104] pnp: PnP ACPI: found 6 devices [ 0.894443] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.897744] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.900047] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.902360] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904826] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.907374] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.910163] NET: Registered protocol family 2 [ 0.912832] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917917] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921944] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927629] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931890] TCP: Hash tables configured (established 65536 bind 65536) [ 0.935203] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938793] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941978] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945250] NET: Registered protocol family 1 [ 0.949367] RPC: Registered named UNIX socket transport module. [ 0.951500] RPC: Registered udp transport module. [ 0.953051] RPC: Registered tcp transport module. [ 0.954767] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956925] NET: Registered protocol family 44 [ 0.958478] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.960567] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962587] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.964798] PCI: CLS 0 bytes, default 64 [ 0.966377] Unpacking initramfs... [ 2.368655] debug: unmapping init [mem 0xffff91517cc54000-0xffff91517ffbffff] [ 2.372947] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.375337] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.378323] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.892943] Initialise system trusted keyrings [ 2.895041] Key type blacklist registered [ 2.897309] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.906974] zbud: loaded [ 2.910465] *** VALIDATE nfs *** [ 2.911812] *** VALIDATE nfs4 *** [ 2.913366] pstore: using deflate compression [ 2.916446] Platform Keyring initialized [ 3.019601] NET: Registered protocol family 38 [ 3.021489] Key type asymmetric registered [ 3.023192] Asymmetric key parser 'x509' registered [ 3.025133] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.028590] io scheduler mq-deadline registered [ 3.030549] io scheduler kyber registered [ 3.032369] io scheduler bfq registered [ 3.033954] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.036483] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.038863] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.042142] ACPI: Power Button [PWRF] [ 3.048135] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.055972] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.068526] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.074537] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.092068] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.122835] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152253] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.157924] Non-volatile memory driver v1.3 [ 3.160140] Linux agpgart interface v0.103 [ 3.192722] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.195667] vda: detected capacity change from 0 to 74694656 [ 3.212728] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.217514] vdb: detected capacity change from 0 to 1073741824 [ 3.232548] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.235771] vdc: detected capacity change from 0 to 2621440000 [ 3.254835] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.257874] vdd: detected capacity change from 0 to 2621440000 [ 3.274600] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.277644] vde: detected capacity change from 0 to 4294967296 [ 3.294268] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.297051] vdf: detected capacity change from 0 to 4294967296 [ 3.305624] libphy: Fixed MDIO Bus: probed [ 3.317678] usbcore: registered new interface driver usbserial_generic [ 3.320566] usbserial: USB Serial support registered for generic [ 3.323424] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.328836] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.331035] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.333833] mousedev: PS/2 mouse device common for all mice [ 3.337222] rtc_cmos 00:05: RTC can wake from S4 [ 3.340994] rtc_cmos 00:05: registered as rtc0 [ 3.343000] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.343065] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.346164] intel_pstate: CPU model not supported [ 3.353032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.358188] hid: raw HID events driver (C) Jiri Kosina [ 3.359759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.360619] usbcore: registered new interface driver usbhid [ 3.367694] usbhid: USB HID core driver [ 3.369443] drop_monitor: Initializing network drop monitor service [ 3.371844] Initializing XFRM netlink socket [ 3.373828] NET: Registered protocol family 10 [ 3.376511] Segment Routing with IPv6 [ 3.377964] NET: Registered protocol family 17 [ 3.380789] mpls_gso: MPLS GSO support [ 3.386759] RAS: Correctable Errors collector initialized. [ 3.389283] AVX version of gcm_enc/dec engaged. [ 3.391180] AES CTR mode by8 optimization enabled [ 3.467759] sched_clock: Marking stable (3467664969, 0)->(4394945811, -927280842) [ 3.471706] registered taskstats version 1 [ 3.474181] Loading compiled-in X.509 certificates [ 3.476755] zswap: loaded using pool lzo/zbud [ 3.501309] Key type big_key registered [ 3.513258] Key type encrypted registered [ 3.515059] ima: No TPM chip found, activating TPM-bypass! [ 3.517670] ima: Allocated hash algorithm: sha1 [ 3.519921] ima: No architecture policies found [ 3.521972] evm: Initialising EVM extended attributes: [ 3.524118] evm: security.selinux [ 3.526019] evm: security.ima [ 3.527132] evm: security.capability [ 3.528729] evm: HMAC attrs: 0x1 [ 3.531227] rtc_cmos 00:05: setting system clock to 2026-08-06 23:28:28 UTC (1786058908) [ 3.538200] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.541694] debug: unmapping init [mem 0xffffffff90982000-0xffffffff90c58fff] [ 3.554099] Write protecting the kernel read-only data: 28672k [ 3.557598] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 3.561263] debug: unmapping init [mem 0xffffffff8f914000-0xffffffff8f9fffff] [ 3.603467] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.617212] systemd[1]: Detected virtualization kvm. [ 3.619390] systemd[1]: Detected architecture x86-64. [ 3.621479] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.645757] systemd[1]: No hostname configured. [ 3.647753] systemd[1]: Set hostname to . [ 3.650265] random: systemd: uninitialized urandom read (16 bytes read) [ 3.653089] systemd[1]: Initializing machine ID from random generator. [ 3.703579] random: ln: uninitialized urandom read (6 bytes read) [ 3.794944] random: systemd: uninitialized urandom read (16 bytes read) [ 3.800099] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.809168] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.815704] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.510309] device-mapper: uevent: version 1.0.3 [ 4.512819] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook.[ 4.761477] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.335675] virtio_net virtio0 ens2: renamed from eth0 [ 5.443571] scsi host0: ata_piix [ 5.496652] scsi host1: ata_piix [ 5.501415] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.504194] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.156518] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.036255] random: crng init done [ 10.037862] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.591107] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.882194] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.141420] SELinux: Disabled at runtime. [ 12.201925] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.211556] systemd[1]: Detected virtualization kvm. [ 12.213710] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.743741] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.747696] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.753977] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.758286] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.762464] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.770774] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.779983] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 12.915447] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.287534] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.652579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.704841] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.817262] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.847426] EDAC sbridge: Ver: 1.1.2 [ 15.525090] Key type dns_resolver registered [ 15.854224] NFS: Registering the id_resolver key type [ 15.856801] Key type id_resolver registered [ 15.858768] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg603-server login: [ 46.790187] hrtimer: interrupt took 3167633 ns [ 63.856297] libcfs: loading out-of-tree module taints kernel. [ 63.893594] Key type ._llcrypt registered [ 63.895834] Key type .llcrypt registered [ 63.987874] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 81.294343] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 82.834467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.853965] alg: No test for adler32 (adler32-zlib) [ 84.152767] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 85.029820] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 86.784087] Key type lgssc registered [ 88.471872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.480106] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.650308] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 150.631118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.655936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.872435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.908749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.997445] Lustre: lustre-MDT0000: new disk, initializing [ 152.094944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.121203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.647860] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.826558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.934843] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.977922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.984124] Lustre: Skipped 1 previous similar message [ 170.098873] Lustre: lustre-MDT0001: new disk, initializing [ 170.189247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.225264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.234515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.502234] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.824299] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.194414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.424507] Lustre: lustre-OST0000: new disk, initializing [ 187.427689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.432870] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.504699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.685800] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.115075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.127225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.184321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.736902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.879689] Lustre: lustre-OST0001: new disk, initializing [ 206.891880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.899293] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.979128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.466689] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.489639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.495052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.547638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.512784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.003619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.211752] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.036585] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing yml_node [ 250.558589] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 253.105630] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 255.987648] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 257.922757] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 19:32:41 EDT 2026 [ 273.786821] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 275.154886] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.347715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.356177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.368326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.369449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.369692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.396046] Lustre: Skipped 1 previous similar message [ 296.169955] Lustre: server umount lustre-MDT0000 complete [ 299.363467] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059204 with bad export cookie 8711012084426223777 [ 299.370229] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.370812] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 299.490216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.508599] Lustre: Skipped 2 previous similar messages [ 299.514810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.524495] Lustre: Skipped 2 previous similar messages [ 304.609164] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.609323] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.624497] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 305.763213] Lustre: server umount lustre-MDT0001 complete [ 309.185983] Lustre: server umount lustre-OST0000 complete [ 312.751898] Lustre: server umount lustre-OST0001 complete [ 319.593762] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 329.161456] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 358.850618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.088737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.131934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.234524] Lustre: lustre-MDT0000: new disk, initializing [ 359.311154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.324696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.540458] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 373.657757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.798316] Lustre: 15065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.859276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.863637] Lustre: Skipped 1 previous similar message [ 373.949732] Lustre: lustre-MDT0001: new disk, initializing [ 374.033555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.056054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.079715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.076957] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 386.366841] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.085085] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 397.510793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 397.910753] Lustre: lustre-OST0000: new disk, initializing [ 397.917739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 397.934987] Lustre: 17032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 398.043221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.175953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 399.190111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 399.282791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.191205] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 412.692452] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 417.024471] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 417.213605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 421.659903] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 421.825744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 424.933885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.940491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.954060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.955325] Lustre: Skipped 2 previous similar messages [ 424.965778] Lustre: Skipped 2 previous similar messages [ 429.614073] Lustre: server umount lustre-OST0000 complete [ 435.169983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.180765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.195854] Lustre: Skipped 1 previous similar message [ 438.690153] Lustre: server umount lustre-MDT0000 complete [ 440.289206] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.309031] LustreError: 15077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 442.193684] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059347 with bad export cookie 8711012084426225310 [ 442.198874] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.201659] LustreError: 15057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 442.564903] Lustre: server umount lustre-MDT0001 complete [ 446.596130] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:35:50 (1786059350) [ 448.831477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 450.473715] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:35:54 (1786059354) [ 456.067063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.531477] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.634744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.015833] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 461.794939] LustreError: 19441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.924324] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.265212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.976183] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 480.272326] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.345567] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.660491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.715701] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 503.625452] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.461198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.473679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.485461] Lustre: Skipped 2 previous similar messages [ 516.865099] Lustre: server umount lustre-OST0000 complete [ 520.674688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.770745] Lustre: server umount lustre-MDT0000 complete [ 526.819267] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.851436] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 530.413968] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059435 with bad export cookie 8711012084426226192 [ 530.419190] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.423253] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 530.731601] Lustre: server umount lustre-MDT0001 complete [ 542.616982] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 545.861033] Key type lgssc unregistered [ 546.130832] LNet: 22973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.145285] LNetError: 22973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.157731] LNet: Removed LNI 192.168.206.103@tcp [ 547.240191] Key type .llcrypt unregistered [ 547.246430] Key type ._llcrypt unregistered [ 568.809931] Key type ._llcrypt registered [ 568.812625] Key type .llcrypt registered [ 568.930970] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 570.212603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.263906] alg: No test for adler32 (adler32-zlib) [ 571.249576] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 571.548049] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 573.207244] Key type lgssc registered [ 574.513272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 582.952557] loop: module loaded [ 583.023448] loop0: detected capacity change from 0 to 204800000 [ 583.264703] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 583.277515] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 583.704596] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.176771] loop0: detected capacity change from 0 to 204800000 [ 589.305114] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.320052] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.336628] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 594.489660] loop0: detected capacity change from 0 to 204800000 [ 600.375363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 600.403560] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.735696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 601.800810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 601.900848] Lustre: lustre-MDT0000: new disk, initializing [ 602.012096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.035325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 606.354374] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 614.076171] loop1: detected capacity change from 0 to 204800000 [ 618.921084] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 619.197407] Lustre: lustre-OST07c6: new disk, initializing [ 619.200607] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 619.204295] Lustre: Skipped 1 previous similar message [ 619.208702] Lustre: 26158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 619.269205] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 624.390965] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 626.785279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 626.795901] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 626.826204] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 629.187840] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 629.196542] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 629.202129] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 629.208042] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 629.213206] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 629.220723] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 638.833798] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 638.850411] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 638.856053] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 638.866263] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 638.874539] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 638.887061] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 643.298178] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 643.309991] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 643.315952] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 643.321187] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 643.325726] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 643.331100] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 643.336928] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 643.346537] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 643.351043] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 643.355636] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 643.360017] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 643.364539] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 646.482388] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 646.490502] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 646.497283] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 646.503459] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 646.507641] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 646.514085] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 646.520569] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 646.530764] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 646.540782] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 646.550924] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 646.556929] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 646.565768] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.843646] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 650.861713] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 650.872766] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 650.883613] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.894775] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 650.904513] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.915384] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 650.924155] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.929626] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 650.933783] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.939510] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 650.943740] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.424917] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 660.435147] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 660.441876] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 660.450365] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 660.456705] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 660.464976] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 660.476557] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 660.484440] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 660.491931] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 660.501550] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 660.510513] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.518536] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 713.696790] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 713.701586] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.714421] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 714.833908] Lustre: server umount lustre-OST07c6 complete [ 723.888051] Lustre: server umount lustre-MDT0000 complete [ 739.301923] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 741.855270] Key type lgssc unregistered [ 742.166867] LNet: 28718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.173507] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.192498] LNet: Removed LNI 192.168.206.103@tcp [ 743.113400] Key type .llcrypt unregistered [ 743.124508] Key type ._llcrypt unregistered [ 759.069964] Key type ._llcrypt registered [ 759.072896] Key type .llcrypt registered [ 759.200318] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 760.147726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.197957] alg: No test for adler32 (adler32-zlib) [ 761.270516] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 761.505747] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 763.184123] Key type lgssc registered [ 764.198858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.775718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.802570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.337391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 776.178083] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 782.817826] LustreError: 29968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.836638] LustreError: 29968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 783.225639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.636180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 787.650241] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 794.306435] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 799.591837] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 805.209700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.487155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.815677] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 817.278533] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 817.780604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 833.505578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.508397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.533073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 837.827777] Lustre: server umount lustre-OST0000 complete [ 843.747100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.750504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 843.761249] Lustre: Skipped 2 previous similar messages [ 843.777430] Lustre: Skipped 2 previous similar messages [ 847.534724] Lustre: server umount lustre-MDT0000 complete [ 848.888533] LustreError: 29969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.915103] LustreError: 29969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 851.080407] LustreError: 29948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059756 with bad export cookie 10369722634156531216 [ 851.097846] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.104984] LustreError: 29948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 851.366470] Lustre: server umount lustre-MDT0001 complete [ 861.611754] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 863.948775] Key type lgssc unregistered [ 864.242482] LNet: 33703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.264591] LNetError: 33703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.287846] LNet: Removed LNI 192.168.206.103@tcp [ 865.027143] Key type .llcrypt unregistered [ 865.030978] Key type ._llcrypt unregistered [ 882.491760] Key type ._llcrypt registered [ 882.495307] Key type .llcrypt registered [ 882.642141] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 883.526859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.651966] alg: No test for adler32 (adler32-zlib) [ 884.790637] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 885.037904] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 886.727180] Key type lgssc registered [ 887.786713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.709364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.738439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.136447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.651541] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 905.286565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.508520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 908.351906] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 913.347603] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.132809] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 922.063304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.316468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.421928] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 933.752415] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 941.804894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 951.265444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.275232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.291160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.292214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.295327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.306231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.331087] Lustre: Skipped 1 previous similar message [ 957.272334] Lustre: server umount lustre-OST0000 complete [ 962.018937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 962.033526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.050071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.387544] Lustre: server umount lustre-MDT0000 complete [ 967.658625] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.671188] LustreError: 34981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 969.199765] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059874 with bad export cookie 13645864245984594192 [ 969.201793] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.212931] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 969.514756] Lustre: server umount lustre-MDT0001 complete [ 978.157506] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 980.746395] Key type lgssc unregistered [ 980.983957] LNet: 38549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.990664] LNetError: 38549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.005774] LNet: Removed LNI 192.168.206.103@tcp [ 981.658487] Key type .llcrypt unregistered [ 981.661872] Key type ._llcrypt unregistered [ 1000.620370] Key type ._llcrypt registered [ 1000.625654] Key type .llcrypt registered [ 1000.726959] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1001.423630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.448465] alg: No test for adler32 (adler32-zlib) [ 1002.499375] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1002.754927] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1004.447435] Key type lgssc registered [ 1005.446964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.666856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1012.692655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.235602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.876369] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1024.481884] LustreError: 39945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.489927] LustreError: 39945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1026.326481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.642743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.998279] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1036.335123] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1041.981993] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1048.717305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.303032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.205596] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1059.582058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1062.380835] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1069.367266] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [ 1070.051289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.052544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1070.068532] Lustre: Skipped 2 previous similar messages [ 1070.077295] Lustre: Skipped 1 previous similar message [ 1074.467144] LustreError: 39941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.864216] Lustre: server umount lustre-MDT0000 complete [ 1078.829372] LustreError: 39925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786059983 with bad export cookie 6231625069047536990 [ 1078.837352] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1078.838386] LustreError: 39925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1079.579616] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.600491] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1079.604689] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.3@tcp (stopping) [ 1079.624738] Lustre: Skipped 1 previous similar message [ 1080.295263] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.309938] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.314158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1083.335410] LustreError: 39939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1083.343910] Lustre: Skipped 1 previous similar message [ 1085.210338] Lustre: server umount lustre-MDT0001 complete [ 1106.159900] Lustre: server umount lustre-OST0000 complete [ 1120.256290] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1122.836607] Key type lgssc unregistered [ 1123.077619] LNet: 43808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.091054] LNetError: 43808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.107885] LNet: Removed LNI 192.168.206.103@tcp [ 1123.916087] Key type .llcrypt unregistered [ 1123.918280] Key type ._llcrypt unregistered [ 1147.644307] Key type ._llcrypt registered [ 1147.646553] Key type .llcrypt registered [ 1147.734302] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1149.039045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.071827] alg: No test for adler32 (adler32-zlib) [ 1150.328270] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1150.534661] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1152.191179] Key type lgssc registered [ 1153.580558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.068723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.083816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.386101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.253597] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1170.462272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.770719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1173.481660] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1179.307899] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1183.947501] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.649128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.954734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.262312] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1201.126295] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1206.316263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1210.336219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.349294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.360471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.366606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.370808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.382793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1216.484468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.491267] Lustre: Skipped 3 previous similar messages [ 1221.602736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.608759] Lustre: Skipped 2 previous similar messages [ 1224.159456] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1224.303315] Lustre: server umount lustre-OST0000 complete [ 1242.083928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.100682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.108150] Lustre: Skipped 1 previous similar message [ 1242.108364] Lustre: Skipped 1 previous similar message [ 1251.298846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.315602] Lustre: Skipped 4 previous similar messages [ 1255.391074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1255.586167] Lustre: server umount lustre-MDT0000 complete [ 1256.418390] LustreError: 45219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.435066] LustreError: 45219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1259.170125] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060164 with bad export cookie 8776588395259205042 [ 1259.187189] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.190572] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1265.539081] Lustre: server umount lustre-MDT0001 complete [ 1276.172437] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1279.087382] Key type lgssc unregistered [ 1279.411246] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.423828] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.448038] LNet: Removed LNI 192.168.206.103@tcp [ 1280.428467] Key type .llcrypt unregistered [ 1280.430268] Key type ._llcrypt unregistered [ 1301.622596] Key type ._llcrypt registered [ 1301.627479] Key type .llcrypt registered [ 1301.741195] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1302.840983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1302.865349] alg: No test for adler32 (adler32-zlib) [ 1303.926776] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1304.124258] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1305.785128] Key type lgssc registered [ 1306.752100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.322329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.360330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.791089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.116140] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1325.029472] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.054547] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1326.872308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.322709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.361041] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1338.746907] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1343.779562] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.968123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.271738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.706100] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1359.562277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1362.849747] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1368.728668] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1371.648635] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1383.642853] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-0619h51m26s [ 1384.892233] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1389.025454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.036352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.056704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.121695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.134740] Lustre: Skipped 4 previous similar messages [ 1391.969735] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [ 1391.975720] Lustre: Skipped 1 previous similar message [ 1392.850871] Lustre: server umount lustre-MDT0000 complete [ 1395.534437] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1396.196510] LustreError: 52863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.206612] LustreError: 52863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1400.663191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.770334] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1400.970278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.025408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1402.034819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1405.218867] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1407.932749] Lustre: DEBUG MARKER: Wait for df (36658) ... [ 1409.012964] Lustre: DEBUG MARKER: done [ 1412.578571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.582061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.587433] Lustre: Skipped 3 previous similar messages [ 1417.698640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.707529] Lustre: Skipped 2 previous similar messages [ 1426.399118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1426.566395] Lustre: server umount lustre-OST0000 complete [ 1433.056788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1433.060972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.064962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.064971] Lustre: Skipped 2 previous similar messages [ 1433.079680] Lustre: Skipped 1 previous similar message [ 1435.944260] Lustre: server umount lustre-MDT0000 complete [ 1439.728930] LustreError: 50330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060344 with bad export cookie 10001650086258878462 [ 1439.732925] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.753095] LustreError: 50330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1440.228302] Lustre: server umount lustre-MDT0001 complete [ 1451.888685] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1454.692398] Key type lgssc unregistered [ 1454.998609] LNet: 55277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.011627] LNetError: 55277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.029205] LNet: Removed LNI 192.168.206.103@tcp [ 1455.821173] Key type .llcrypt unregistered [ 1455.824364] Key type ._llcrypt unregistered [ 1490.290970] Key type ._llcrypt registered [ 1490.294275] Key type .llcrypt registered [ 1490.373970] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1491.315268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.509826] alg: No test for adler32 (adler32-zlib) [ 1492.536953] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1492.766347] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1494.431195] Key type lgssc registered [ 1495.529226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1502.306406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1502.329212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.585944] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1503.645505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1503.809353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.460253] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1514.105670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.187307] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1514.218921] Lustre: 57485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1514.443705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1517.891461] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1523.440512] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1527.689620] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1532.704553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.839570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1532.995930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.389816] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1538.036748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1542.524513] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1548.686790] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-0619h54m12s [ 1549.802672] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1555.380724] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1557.451781] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1557.496066] LustreError: 58598:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1577.814970] Lustre: lustre-MDT0000: Client dc0ed507-5ee4-424e-8a56-54f2f15588d4 (at 192.168.206.3@tcp) reconnecting [ 1597.511141] LustreError: 58598:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1599.682091] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1602.094352] Lustre: DEBUG MARKER: done [ 1609.717871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.720244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.722818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.757876] Lustre: Skipped 1 previous similar message [ 1610.116154] Lustre: server umount lustre-OST0000 complete [ 1614.824817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.836627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.843437] Lustre: Skipped 1 previous similar message [ 1614.856470] Lustre: Skipped 2 previous similar messages [ 1615.908922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.928740] Lustre: Skipped 1 previous similar message [ 1619.960386] Lustre: server umount lustre-MDT0000 complete [ 1623.127815] LustreError: 57477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060528 with bad export cookie 136573303144754567 [ 1623.140607] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.480819] Lustre: server umount lustre-MDT0001 complete [ 1633.185292] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1635.730109] Key type lgssc unregistered [ 1635.989939] LNet: 61811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.997887] LNetError: 61811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.030286] LNet: Removed LNI 192.168.206.103@tcp [ 1637.726203] Key type .llcrypt unregistered [ 1637.731107] Key type ._llcrypt unregistered [ 1675.894881] Key type ._llcrypt registered [ 1675.897170] Key type .llcrypt registered [ 1676.035919] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1676.871975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.896852] alg: No test for adler32 (adler32-zlib) [ 1678.019901] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1678.287250] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1679.951197] Key type lgssc registered [ 1680.928922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.550257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1683.572550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.788449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1684.824680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1684.968967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.661549] Lustre: Failing over lustre-MDT0000 [ 1685.921782] Lustre: server umount lustre-MDT0000 complete [ 1690.830735] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:56:34 (1786060594) [ 1696.271733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.735623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.448396] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1707.548329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.619661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1707.653528] Lustre: 64588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1707.857639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1711.861447] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1718.016976] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1723.114510] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1729.110415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.352911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1729.536172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.355309] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1737.892268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1744.569604] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1750.481251] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1758.688090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.690768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.702592] Lustre: Skipped 2 previous similar messages [ 1758.717923] Lustre: Skipped 2 previous similar messages [ 1762.055800] Lustre: server umount lustre-MDT0000 complete [ 1765.889240] LustreError: 64580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060670 with bad export cookie 15130449455997073207 [ 1765.893300] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.922813] LustreError: 64580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1766.165839] Lustre: server umount lustre-MDT0001 complete [ 1769.466431] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1773.130582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1781.021641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.287208] Lustre: 63514:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060673/real 1786060673] req@0000000047525b98 x1872820261920128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1784.333832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.335141] Lustre: 63513:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060675/real 1786060675] req@000000007e4e18af x1872820261920256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1789.409387] Lustre: 63515:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060678/real 1786060678] req@00000000750ebf1d x1872820261920384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1790.830037] LustreError: 66436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.849130] LustreError: 66436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1790.961426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.057265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1791.462903] Lustre: 63516:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060680/real 1786060680] req@0000000040de6f45 x1872820261920640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060696 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1793.955401] LustreError: 66437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.981229] LustreError: 68357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.006260] LustreError: 68357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1795.251792] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1797.025174] Lustre: 63513:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060685/real 1786060685] req@00000000d3ebd96b x1872820261920768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1800.162534] LustreError: 68356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.184161] LustreError: 68356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1803.473047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.880578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.269497] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1809.393115] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1813.555223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1815.189459] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.733231] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1828.836726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.838885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.855545] Lustre: Skipped 2 previous similar messages [ 1828.873774] Lustre: Skipped 2 previous similar messages [ 1832.135981] Lustre: server umount lustre-MDT0000 complete [ 1833.953425] LustreError: 68355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.970938] LustreError: 68355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1835.573303] LustreError: 64580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060740 with bad export cookie 15130449455997075489 [ 1835.580962] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.582413] LustreError: 64580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1835.799319] Lustre: server umount lustre-MDT0001 complete [ 1841.925261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1850.343582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.456599] Lustre: 63516:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786060744/real 1786060744] req@00000000fb577434 x1872820261946880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786060760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1855.488777] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.958676] LustreError: 69093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.047352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.113720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1866.146715] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1874.700115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.274131] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 1880.563595] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1884.732874] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1886.299896] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1891.633419] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1899.437341] Lustre: server umount lustre-MDT0000 complete [ 1900.013924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.027380] LustreError: 71040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.032911] Lustre: Skipped 2 previous similar messages [ 1900.049697] LustreError: 71040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 1902.983900] LustreError: 64581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060807 with bad export cookie 15130449455997076868 [ 1902.992513] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.999993] LustreError: 64581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1903.310791] Lustre: server umount lustre-MDT0001 complete [ 1908.819547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1912.048326] Lustre: DEBUG MARKER: files compared the same [ 1923.191566] Lustre: server umount lustre-OST0000 complete [ 1938.135192] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 1940.281425] Key type lgssc unregistered [ 1940.585700] LNet: 74315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.593697] LNetError: 74315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.608776] LNet: Removed LNI 192.168.206.103@tcp [ 1941.356938] Key type .llcrypt unregistered [ 1941.358067] Key type ._llcrypt unregistered [ 1964.629519] Key type ._llcrypt registered [ 1964.631606] Key type .llcrypt registered [ 1964.712389] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 1965.609221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.636096] alg: No test for adler32 (adler32-zlib) [ 1966.660385] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1966.946864] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 1968.639426] Key type lgssc registered [ 1969.366410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.301632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.344885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.777656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.668544] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug +malloc all [ 1987.039785] LustreError: 75744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.062062] LustreError: 75744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1990.192035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.811373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.395417] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug +malloc all [ 2004.426201] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2010.853242] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.161558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2019.772722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.891106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2025.731277] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug +malloc all [ 2032.949404] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.362727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.371047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.386327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.409363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.411820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.431945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.791557] Lustre: server umount lustre-OST0000 complete [ 2052.063918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2052.070248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.116632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.130137] Lustre: Skipped 1 previous similar message [ 2055.648657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.650737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.771614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.777337] Lustre: Skipped 2 previous similar messages [ 2065.376103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2065.635568] Lustre: server umount lustre-MDT0000 complete [ 2065.893574] LustreError: 75745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.912797] LustreError: 75745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2070.071861] LustreError: 75724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786060975 with bad export cookie 2518744730262478201 [ 2070.089217] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.614391] Lustre: server umount lustre-MDT0001 complete [ 2086.006580] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2089.242159] Key type lgssc unregistered [ 2089.580044] LNet: 79280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.591536] LNetError: 79280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.602761] LNet: Removed LNI 192.168.206.103@tcp [ 2090.371691] Key type .llcrypt unregistered [ 2090.375795] Key type ._llcrypt unregistered [ 2111.286976] Key type ._llcrypt registered [ 2111.290384] Key type .llcrypt registered [ 2111.401565] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2112.250914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.300344] alg: No test for adler32 (adler32-zlib) [ 2113.337196] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2113.557488] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2115.231197] Key type lgssc registered [ 2116.200827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2123.048545] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2123.075300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2138.656117] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2143.775328] LustreError: 80663:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.103@tcp: failed processing log, type 4: rc = -110 [ 2170.399220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.450589] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2186.379474] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.127264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.213404] Lustre: *** cfs_fail_loc=706, val=0*** [ 2194.537197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.600759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2197.516321] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2200.037631] LustreError: 80689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.066705] LustreError: 80689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2205.057727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.160176] LustreError: 80688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.189090] LustreError: 80688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2205.495795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.866324] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2217.033119] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2222.316189] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2225.121955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.136071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.154905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.144646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2226.156040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.156056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.457103] Lustre: server umount lustre-OST0000 complete [ 2236.388754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.390450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.390559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.406120] Lustre: Skipped 1 previous similar message [ 2236.411195] Lustre: Skipped 1 previous similar message [ 2240.558346] Lustre: server umount lustre-MDT0000 complete [ 2241.519208] LustreError: 81754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.538378] LustreError: 81754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2244.168111] LustreError: 80669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786061149 with bad export cookie 7972294301039766162 [ 2244.181124] LustreError: 80669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2244.184090] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.554621] Lustre: server umount lustre-MDT0001 complete [ 2255.116799] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2257.784530] Key type lgssc unregistered [ 2258.116436] LNet: 84279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.130850] LNetError: 84279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.145861] LNet: Removed LNI 192.168.206.103@tcp [ 2259.018629] Key type .llcrypt unregistered [ 2259.020394] Key type ._llcrypt unregistered [ 2284.485881] Key type ._llcrypt registered [ 2284.488674] Key type .llcrypt registered [ 2284.626314] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2285.749843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.892291] alg: No test for adler32 (adler32-zlib) [ 2286.938806] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2287.131956] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2288.823176] Key type lgssc registered [ 2289.748598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.217929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.230793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.778847] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2310.677192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.285279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.745140] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2316.259640] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.274065] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2321.377525] LustreError: 86384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.398703] LustreError: 86384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2323.554231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2323.989442] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.005673] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2324.014856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.058271] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.089707] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2330.139208] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2334.181126] LustreError: 86385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.205612] LustreError: 86385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2338.911329] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.954235] Lustre: MGS: Client b8272e2e-7a60-4566-8c64-82cea49b3743 (at 0@lo) reconnecting [ 2338.973475] LustreError: 86400:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2339.374901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.503556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2345.268203] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2359.781278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.787481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.793212] Lustre: Skipped 1 previous similar message [ 2359.798065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2364.898584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2364.908905] Lustre: Skipped 2 previous similar messages [ 2365.055374] Lustre: server umount lustre-OST0000 complete [ 2370.022862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.033816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.038366] Lustre: Skipped 1 previous similar message [ 2374.631897] Lustre: server umount lustre-MDT0000 complete [ 2375.139084] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.159457] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2378.529843] Lustre: server umount lustre-MDT0001 complete [ 2381.917532] Lustre: server umount MGS complete [ 2392.285124] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:08:15 (1786061295) [ 2419.800371] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 2429.895474] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2460.279459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.746499] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2473.456173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.694391] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2473.766122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2473.883723] Lustre: lustre-MDT0001: new disk, initializing [ 2473.965825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.998951] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2474.013860] Lustre: 93564:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2474.028891] LustreError: 93564:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2478.468298] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2488.442202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2488.686878] Lustre: lustre-OST0000: new disk, initializing [ 2488.706738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2488.716278] Lustre: 94330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2488.789658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.959439] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2499.692808] Lustre: 94901:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000303714c0 x1872820900503808/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786061420 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2501.691678] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2506.336766] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.371749] Lustre: MGS: Client 667f9acf-da8c-4e79-a6f7-12e302fd7a08 (at 0@lo) reconnecting [ 2506.432836] Lustre: 92833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2506.444884] LustreError: 93550:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2506.461380] LustreError: 93535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.476554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2506.493154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2506.534834] Lustre: lustre-MDT0000: new disk, initializing [ 2506.691121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.710472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2508.853767] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -11 [ 2510.568768] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2511.857305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2511.864299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2512.947064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2512.956934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2512.983988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2537.443964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.449603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.464895] Lustre: Skipped 2 previous similar messages [ 2537.474892] Lustre: Skipped 1 previous similar message [ 2541.367333] Lustre: server umount lustre-OST0000 complete [ 2547.680509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.712477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.714666] Lustre: Skipped 1 previous similar message [ 2550.923797] Lustre: server umount lustre-MDT0000 complete [ 2552.802813] LustreError: 93540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.496217] Lustre: server umount lustre-MDT0001 complete [ 2564.765960] Lustre: server umount MGS complete [ 2577.367796] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:11:20 (1786061480) [ 2595.825590] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2599.540983] Key type lgssc unregistered [ 2599.964936] LNet: 97902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2599.971113] LNetError: 97902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2599.979760] LNet: Removed LNI 192.168.206.103@tcp [ 2600.953972] Key type .llcrypt unregistered [ 2600.958315] Key type ._llcrypt unregistered [ 2626.013198] Key type ._llcrypt registered [ 2626.019361] Key type .llcrypt registered [ 2626.191985] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2627.796367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2627.954039] alg: No test for adler32 (adler32-zlib) [ 2629.416638] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2629.862115] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2631.743459] Key type lgssc registered [ 2632.805986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2639.364690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2639.393979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.790280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.236694] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2650.084872] LustreError: 99125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.114756] LustreError: 99125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2652.359489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.710853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2658.066210] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2666.324450] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2673.150934] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2679.178044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.493725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.730526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2686.421315] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2695.068269] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2706.401411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2706.416711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.450385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2708.450560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2708.452814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2708.470991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.904278] Lustre: server umount lustre-OST0000 complete [ 2718.690400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.692742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.715137] Lustre: Skipped 1 previous similar message [ 2718.728984] Lustre: Skipped 1 previous similar message [ 2722.285632] Lustre: server umount lustre-MDT0000 complete [ 2723.812871] LustreError: 99119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.837351] LustreError: 99119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2726.301409] LustreError: 101909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786061631 with bad export cookie 15313163599511021381 [ 2726.310713] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.322613] LustreError: 101909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2726.656660] Lustre: server umount lustre-MDT0001 complete [ 2736.875540] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2740.090282] Key type lgssc unregistered [ 2740.514660] LNet: 102663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2740.522232] LNetError: 102663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2740.544066] LNet: Removed LNI 192.168.206.103@tcp [ 2742.021618] Key type .llcrypt unregistered [ 2742.024094] Key type ._llcrypt unregistered [ 2758.840701] Key type ._llcrypt registered [ 2758.843422] Key type .llcrypt registered [ 2758.929344] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2759.832601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.869914] alg: No test for adler32 (adler32-zlib) [ 2760.891645] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2761.103005] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2762.799418] Key type lgssc registered [ 2763.822470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2769.161196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2769.171374] LustreError: 103749:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2769.174332] LustreError: 103752:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2769.186242] LustreError: 103749:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2769.195055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.195581] LustreError: 103749:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2769.195598] LustreError: 103749:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2770.689249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.039815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.387983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.486677] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2791.080727] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing lsmod [ 2793.707562] LustreError: 104957:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2793.727135] LustreError: 104958:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2793.733607] LustreError: 104957:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 2793.741525] LustreError: 104957:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2793.746995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.753788] LustreError: 104957:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2794.222698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.246261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 2800.101027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.111518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.125979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.612030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.625835] Lustre: Skipped 1 previous similar message [ 2800.635740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.647920] Lustre: Skipped 1 previous similar message [ 2803.179156] Lustre: server umount lustre-MDT0000 complete [ 2804.781603] LustreError: 103784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.803976] LustreError: 103784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2806.476394] LustreError: 103765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786061711 with bad export cookie 11407047840456735678 [ 2806.479923] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.495792] LustreError: 103765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2806.880160] LustreError: 104219:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb29e60a x1872821397508480/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2807.115160] Lustre: server umount lustre-MDT0001 complete [ 2821.091245] Lustre: server umount lustre-OST0000 complete [ 2829.249360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.700798] LustreError: 106020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.830199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.079882] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2834.913174] LustreError: 106021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.047109] LustreError: 106020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.127918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.593312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2848.334840] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2855.596596] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2861.117459] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2867.064097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.428860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.554925] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2875.635451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 2880.526354] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2888.482568] Lustre: server umount lustre-OST0000 complete [ 2894.823850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.837858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.861872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.474121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.490729] Lustre: Skipped 3 previous similar messages [ 2901.548508] Lustre: server umount lustre-MDT0000 complete [ 2904.627213] LustreError: 106003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786061809 with bad export cookie 11407047840456736595 [ 2904.631136] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.638757] LustreError: 106003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2914.907411] Lustre: server umount lustre-MDT0001 complete [ 2924.215774] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 2927.072887] Key type lgssc unregistered [ 2927.358325] LNet: 109555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2927.364435] LNetError: 109555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2927.400833] LNet: Removed LNI 192.168.206.103@tcp [ 2928.085209] Key type .llcrypt unregistered [ 2928.087310] Key type ._llcrypt unregistered [ 2947.243350] Key type ._llcrypt registered [ 2947.245974] Key type .llcrypt registered [ 2947.327127] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 2948.078548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2948.142138] alg: No test for adler32 (adler32-zlib) [ 2949.119897] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2949.309082] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 2950.961445] Key type lgssc registered [ 2951.853429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.358318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.369576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.681450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.164466] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2969.060713] LustreError: 110952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.088329] LustreError: 110952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2969.590944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.873441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.230469] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 2978.890996] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2983.451717] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2988.614320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.916127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.045493] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3000.067709] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3004.277101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3009.488762] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3011.735203] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3014.625493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3014.631124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.651365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.654028] Lustre: Skipped 1 previous similar message [ 3016.692145] Lustre: server umount lustre-OST0000 complete [ 3019.747353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.753953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.758430] Lustre: Skipped 2 previous similar messages [ 3019.770595] Lustre: Skipped 1 previous similar message [ 3024.865586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.874615] Lustre: Skipped 1 previous similar message [ 3025.910792] Lustre: server umount lustre-MDT0000 complete [ 3029.208245] LustreError: 110931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786061934 with bad export cookie 16540396022787723899 [ 3029.221450] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.511940] Lustre: server umount lustre-MDT0001 complete [ 3040.150625] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 3042.741210] Key type lgssc unregistered [ 3043.136847] LNet: 114587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.141884] LNetError: 114587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.154571] LNet: Removed LNI 192.168.206.103@tcp [ 3044.058891] Key type .llcrypt unregistered [ 3044.067315] Key type ._llcrypt unregistered [ 3058.354514] Key type ._llcrypt registered [ 3058.358924] Key type .llcrypt registered [ 3058.433066] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3059.252251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3059.269822] alg: No test for adler32 (adler32-zlib) [ 3060.344190] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3060.523889] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 3062.175160] Key type lgssc registered [ 3063.227727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.729749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3068.746823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.031422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.463482] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3079.144864] LustreError: 115789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.157454] LustreError: 115789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3080.412763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.713653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.497398] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3091.294616] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3095.584235] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3100.192272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.441581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.164658] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3114.890411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3117.242953] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3126.762394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3126.780246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.825151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.337931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.343929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.460335] Lustre: server umount lustre-OST0000 complete [ 3140.578478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.599952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.614708] Lustre: Skipped 2 previous similar messages [ 3143.833869] Lustre: server umount lustre-MDT0000 complete [ 3145.698208] LustreError: 115790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.728682] LustreError: 115790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3147.525537] LustreError: 115771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062052 with bad export cookie 14147009773850120371 [ 3147.539069] LustreError: 115771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3147.541407] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.991355] Lustre: server umount lustre-MDT0001 complete [ 3158.260545] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 3160.769352] Key type lgssc unregistered [ 3161.064332] LNet: 119325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3161.100781] LNetError: 119325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.151942] LNet: Removed LNI 192.168.206.103@tcp [ 3162.957394] Key type .llcrypt unregistered [ 3162.959214] Key type ._llcrypt unregistered [ 3190.174949] Key type ._llcrypt registered [ 3190.177322] Key type .llcrypt registered [ 3190.345608] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3191.965456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3191.995802] alg: No test for adler32 (adler32-zlib) [ 3193.144610] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3193.588261] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 3195.265780] Key type lgssc registered [ 3196.611929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3204.153765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3204.191645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.516365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.538454] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3215.566847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.939927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3219.177737] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3226.516549] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3233.187225] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3238.804866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.041858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.565200] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3249.992755] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3250.348524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3256.360123] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3258.142634] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3260.143709] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3260.150365] Lustre: Skipped 2 previous similar messages [ 3264.665068] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3264.676494] Lustre: Skipped 2 previous similar messages [ 3265.682394] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3265.721454] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3265.734792] Lustre: Skipped 2 previous similar messages [ 3276.553061] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3276.560743] Lustre: Skipped 2 previous similar messages [ 3283.072786] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.3@tcp 192.168.0.[2,10]@tcp [ 3287.434908] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.3/24@tcp in log lustre-MDT0000 [ 3287.441875] Lustre: Skipped 2 previous similar messages [ 3291.118214] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.3/24@tcp [ 3291.135780] Lustre: Skipped 1 previous similar message [ 3301.449177] Lustre: server umount lustre-OST0000 complete [ 3306.984692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.995597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.003805] Lustre: Skipped 1 previous similar message [ 3307.021909] Lustre: Skipped 1 previous similar message [ 3312.096868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.107467] Lustre: Skipped 1 previous similar message [ 3317.215177] Lustre: 120054:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062206/real 1786062206] req@00000000532f1a43 x1872821850535168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786062222 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3317.221439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3317.237755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.238569] Lustre: Skipped 1 previous similar message [ 3319.263644] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3319.480971] Lustre: server umount lustre-MDT0000 complete [ 3322.337342] LustreError: 120744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.338040] Lustre: 120053:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062211/real 1786062211] req@00000000f3bf1474 x1872821850537472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786062227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3322.347470] LustreError: 120744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3322.402352] LustreError: 120723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062227 with bad export cookie 13846838773013345252 [ 3322.404440] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.412384] LustreError: 120723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3322.680600] Lustre: server umount lustre-MDT0001 complete [ 3331.134990] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 3333.390994] Key type lgssc unregistered [ 3333.651270] LNet: 125550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3333.660951] LNetError: 125550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3333.673758] LNet: Removed LNI 192.168.206.103@tcp [ 3334.345731] Key type .llcrypt unregistered [ 3334.352846] Key type ._llcrypt unregistered [ 3356.284602] Key type ._llcrypt registered [ 3356.286417] Key type .llcrypt registered [ 3356.376647] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3356.979520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.055166] alg: No test for adler32 (adler32-zlib) [ 3358.065476] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3358.238538] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 3359.871135] Key type lgssc registered [ 3360.983301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.618303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.666842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.419691] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3370.449258] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.3/24@tcp [ 3370.480686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.698959] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3380.705229] LustreError: 127092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.756844] LustreError: 127092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3385.824349] LustreError: 127091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.737516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.050542] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3387.082256] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.3/24@tcp [ 3387.119383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.848666] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3398.247051] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3403.560363] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3409.460656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.054684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.263887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 3417.235749] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3423.825184] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3436.013058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.013578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.019799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.027910] Lustre: Skipped 1 previous similar message [ 3438.684256] Lustre: server umount lustre-OST0000 complete [ 3443.681778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3443.690673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.706618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.719274] Lustre: Skipped 1 previous similar message [ 3448.751033] Lustre: server umount lustre-MDT0000 complete [ 3449.320238] LustreError: 127087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.333925] LustreError: 127087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3453.214534] LustreError: 127071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062358 with bad export cookie 2271625489069717504 [ 3453.218820] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.040888] Lustre: server umount lustre-MDT0001 complete [ 3464.594148] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 3467.962834] Key type lgssc unregistered [ 3468.375046] LNet: 130684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3468.388795] LNetError: 130684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3468.404601] LNet: Removed LNI 192.168.206.103@tcp [ 3469.394932] Key type .llcrypt unregistered [ 3469.397396] Key type ._llcrypt unregistered [ 3497.350782] Key type ._llcrypt registered [ 3497.353157] Key type .llcrypt registered [ 3497.480521] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 3515.132371] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3516.741753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3516.789643] alg: No test for adler32 (adler32-zlib) [ 3518.006780] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3518.391982] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 3520.231146] Key type lgssc registered [ 3521.839751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.084910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.145334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.498096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.534956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.643499] Lustre: lustre-MDT0000: new disk, initializing [ 3556.785984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.819534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.708523] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3569.746385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.818681] Lustre: 134086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3569.874376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3569.881625] Lustre: Skipped 1 previous similar message [ 3569.975715] Lustre: lustre-MDT0001: new disk, initializing [ 3570.084661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3570.132228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3570.146710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3574.472976] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3584.163338] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3589.705730] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3595.632909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3595.838250] Lustre: lustre-OST0000: new disk, initializing [ 3595.841872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3595.847853] Lustre: 136054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3595.934207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.194464] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3603.549425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3603.567033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3603.681091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3610.793668] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.842913] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3615.013907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3618.796445] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3619.035780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3624.421496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.422652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3624.423035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.437537] Lustre: Skipped 1 previous similar message [ 3624.459269] Lustre: Skipped 1 previous similar message [ 3626.901064] Lustre: server umount lustre-OST0000 complete [ 3631.584322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3631.595636] LustreError: Skipped 1 previous similar message [ 3631.600045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.611341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.662090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.664701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.703506] Lustre: Skipped 1 previous similar message [ 3636.633286] Lustre: server umount lustre-MDT0000 complete [ 3640.092751] LustreError: 134076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062545 with bad export cookie 472573810693334634 [ 3640.096483] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3640.100351] LustreError: 134076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3640.434459] Lustre: server umount lustre-MDT0001 complete [ 3647.940874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.365705] LustreError: 138270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.392713] LustreError: 138270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3648.461042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.231943] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3653.606111] LustreError: 138271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3658.530925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.792576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3662.404925] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3668.706697] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.301423] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3679.515554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.834463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.138657] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3695.127903] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3699.655315] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3699.888585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3705.043350] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3705.197882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3713.185899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3713.357354] Lustre: lustre-OST0001: new disk, initializing [ 3713.362666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3713.367878] Lustre: 141640:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3713.459684] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3719.235423] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3722.938519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3722.965385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3723.064882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3726.952962] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3727.141367] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3731.042739] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3731.200867] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3735.502235] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa0c4de4800.ost_server_uuid 50 [ 3736.777743] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aa0c4de4800.ost_server_uuid in FULL state after 0 sec [ 3748.840949] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.844536] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.872165] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3748.881980] Lustre: Skipped 1 previous similar message [ 3750.630854] Lustre: server umount lustre-OST0001 complete [ 3753.963040] LustreError: 140925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.979836] LustreError: 140925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3754.980683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.990788] LustreError: Skipped 1 previous similar message [ 3754.993732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.007212] Lustre: Skipped 1 previous similar message [ 3755.011060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.079291] LustreError: 142997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.097087] LustreError: 142997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3760.285820] Lustre: server umount lustre-OST0000 complete [ 3764.206473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.220431] Lustre: Skipped 1 previous similar message [ 3764.226568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.242023] Lustre: Skipped 4 previous similar messages [ 3769.730302] Lustre: server umount lustre-MDT0000 complete [ 3772.545767] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062677 with bad export cookie 472573810693335516 [ 3772.548979] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.555782] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3772.835832] Lustre: server umount lustre-MDT0001 complete [ 3788.868479] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 3791.515070] Key type lgssc unregistered [ 3791.793100] LNet: 144643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3791.801359] LNetError: 144643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3792.869588] LNet: Removed LNI 192.168.206.103@tcp [ 3793.851291] Key type .llcrypt unregistered [ 3793.856733] Key type ._llcrypt unregistered [ 3824.196204] Key type ._llcrypt registered [ 3824.197788] Key type .llcrypt registered [ 3824.270868] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 3834.396244] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 3835.302119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3835.322836] alg: No test for adler32 (adler32-zlib) [ 3836.261944] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 3836.449305] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 3838.103149] Key type lgssc registered [ 3839.197037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3868.626848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3868.650213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.794285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3869.815177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3869.880656] Lustre: lustre-MDT0000: new disk, initializing [ 3869.985566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.027423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3873.848959] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3882.211330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.321693] Lustre: 148776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3882.403643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3882.410271] Lustre: Skipped 1 previous similar message [ 3882.503268] Lustre: lustre-MDT0001: new disk, initializing [ 3882.553844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.574345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3882.578168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3886.082488] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3895.073942] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3899.512370] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3905.136216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.384786] Lustre: lustre-OST0000: new disk, initializing [ 3905.388549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3905.395354] Lustre: 150745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3905.459668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.703461] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3913.746626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3913.752732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3913.850698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3918.003990] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3921.341360] Lustre: Failing over lustre-OST0000 [ 3921.488530] Lustre: server umount lustre-OST0000 complete [ 3937.751182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.959570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.984294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.006202] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3939.039256] Lustre: 147000:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062828/real 1786062828] req@00000000de477b47 x1872822524738304/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786062844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3939.791102] Lustre: 147003:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062828/real 1786062828] req@00000000e5df1770 x1872822524738176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786062844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3939.823032] Lustre: 147003:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3940.167831] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3941.027122] Lustre: 147001:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062829/real 1786062829] req@000000007521ee7a x1872822524738816/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786062845 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3942.715544] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3943.430489] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3943.430630] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 3945.504339] Lustre: 147002:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062834/real 1786062834] req@0000000082766503 x1872822524739712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786062850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3945.842647] Lustre: Failing over lustre-MDT0000 [ 3946.025295] Lustre: server umount lustre-MDT0000 complete [ 3948.517888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.531580] Lustre: Skipped 3 previous similar messages [ 3950.623180] Lustre: 147000:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062839/real 1786062839] req@00000000bd2a2360 x1872822524740864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786062855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3953.635893] LustreError: 148790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.657878] LustreError: 148790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3955.484939] LustreError: 148784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.766234] LustreError: 148785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.792806] LustreError: 148785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3962.711727] LustreError: 148785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.732124] LustreError: 148785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3963.047118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.152252] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.372707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.422381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3967.792160] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3967.994480] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 3968.491474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3968.495563] Lustre: Skipped 1 previous similar message [ 3968.522845] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3968.553376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3978.727614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.732392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3978.748951] Lustre: Skipped 1 previous similar message [ 3978.763923] Lustre: Skipped 1 previous similar message [ 3981.279277] Lustre: server umount lustre-OST0000 complete [ 3988.963466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.968112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.968478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.991263] Lustre: Skipped 1 previous similar message [ 3993.097961] Lustre: server umount lustre-MDT0000 complete [ 3994.088560] LustreError: 148790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.120180] LustreError: 148790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3998.707064] LustreError: 150744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062903 with bad export cookie 2812115746309746501 [ 3998.716323] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3998.738798] LustreError: 150744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3999.651748] Lustre: server umount lustre-MDT0001 complete [ 4011.405300] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 4013.552756] Key type lgssc unregistered [ 4013.909639] LNet: 154397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4013.916407] LNetError: 154397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4013.936949] LNet: Removed LNI 192.168.206.103@tcp [ 4014.857096] Key type .llcrypt unregistered [ 4014.860403] Key type ._llcrypt unregistered [ 4052.738370] Key type ._llcrypt registered [ 4052.742709] Key type .llcrypt registered [ 4052.896724] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4054.392422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4054.422363] alg: No test for adler32 (adler32-zlib) [ 4055.423852] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 4055.648768] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 4057.415232] Key type lgssc registered [ 4058.858132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4067.047483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.057214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.354626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.511703] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4078.571134] LustreError: 155929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.616680] LustreError: 155929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4083.679771] LustreError: 155928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.217108] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4088.802353] LustreError: 155929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.876782] LustreError: 155958:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4089.891833] LustreError: 155958:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4089.902058] LustreError: 155958:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4090.343798] Lustre: server umount lustre-MDT0000 complete [ 4104.538381] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 4115.234077] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4148.701638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.967483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.992000] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4149.093949] Lustre: lustre-MDT0000: new disk, initializing [ 4149.176240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.195831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4153.591203] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4163.905043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.034400] Lustre: 159263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4164.116711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4164.119685] Lustre: Skipped 1 previous similar message [ 4164.169607] Lustre: lustre-MDT0001: new disk, initializing [ 4164.255068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4164.288101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4164.305838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.439472] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4178.415365] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4183.954531] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.290129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.638830] Lustre: lustre-OST0000: new disk, initializing [ 4191.643394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4191.655339] Lustre: 161234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4191.789662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.953937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4197.963250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4198.085822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4198.578066] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4207.737536] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.499935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4213.028306] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:38:36 (1786063116) [ 4441.056299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4441.064356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.089473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.128644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.128795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.161993] Lustre: Skipped 2 previous similar messages [ 4447.028718] Lustre: server umount lustre-MDT0000 complete [ 4449.251595] LustreError: 159269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.264233] LustreError: 159269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4450.091419] LustreError: 159255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063355 with bad export cookie 18389797529186721846 [ 4450.094598] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.096845] LustreError: 159255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4454.369106] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.370257] LustreError: 159270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.382394] Lustre: Skipped 1 previous similar message [ 4454.386181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4456.572662] Lustre: server umount lustre-MDT0001 complete [ 4474.335125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4474.435979] Lustre: server umount lustre-OST0000 complete [ 4482.107609] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 4489.706966] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4518.156596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.377386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4518.401769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4518.477630] Lustre: lustre-MDT0000: new disk, initializing [ 4518.632468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.647541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4522.677329] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4532.854219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.962650] Lustre: 165891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4533.024732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4533.030707] Lustre: Skipped 1 previous similar message [ 4533.160808] Lustre: lustre-MDT0001: new disk, initializing [ 4533.264169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4533.314085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4533.320778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4537.884278] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4546.136422] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.306533] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.202431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.470872] Lustre: lustre-OST0000: new disk, initializing [ 4557.505286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4557.513464] Lustre: 167859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4557.583896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.647395] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4565.054806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4565.063242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4565.131364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4570.939763] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4575.088513] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4575.330894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4579.214264] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4579.399468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4580.832177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.833690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.836408] LustreError: Skipped 1 previous similar message [ 4580.855391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.860053] Lustre: Skipped 2 previous similar messages [ 4585.455616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4585.460462] Lustre: Skipped 1 previous similar message [ 4595.167129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4595.319089] Lustre: server umount lustre-OST0000 complete [ 4599.777123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.788670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.806488] Lustre: Skipped 1 previous similar message [ 4599.822200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.824890] Lustre: Skipped 2 previous similar messages [ 4605.923499] LustreError: 165901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.941923] LustreError: 165901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4606.829844] Lustre: server umount lustre-MDT0000 complete [ 4610.496548] LustreError: 165886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063515 with bad export cookie 18389797529186880669 [ 4610.504594] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.895543] Lustre: server umount lustre-MDT0001 complete [ 4619.703912] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:45:23 (1786063523) [ 4634.145722] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 4642.326407] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4672.681988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.939453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4672.960392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4673.027441] Lustre: lustre-MDT0000: new disk, initializing [ 4673.092352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.113443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4677.463537] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4686.289897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.426407] Lustre: 172617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4686.748888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4690.818208] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4698.818569] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4703.633283] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4708.913959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.144137] Lustre: lustre-OST0000: new disk, initializing [ 4709.146692] Lustre: Skipped 1 previous similar message [ 4709.150469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4709.153245] Lustre: Skipped 2 previous similar messages [ 4709.156705] Lustre: 174592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4714.391586] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4715.043186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4715.053827] Lustre: Skipped 1 previous similar message [ 4715.058146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4715.147897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4723.463819] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4730.340078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4730.350108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.357979] Lustre: Skipped 1 previous similar message [ 4730.360607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.364093] Lustre: Skipped 2 previous similar messages [ 4736.362345] Lustre: server umount lustre-OST0000 complete [ 4754.401579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4754.706972] Lustre: server umount lustre-MDT0000 complete [ 4756.450440] LustreError: 172628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.477989] LustreError: 172628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4758.292833] LustreError: 172609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063663 with bad export cookie 18389797529186881558 [ 4758.294519] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4758.305228] LustreError: 172609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4758.846707] Lustre: server umount lustre-MDT0001 complete [ 4766.078252] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:47:49 (1786063669) [ 4779.476713] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 4787.230485] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4816.714638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.961920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.986485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4817.053726] Lustre: lustre-MDT0000: new disk, initializing [ 4817.270819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.280608] Lustre: Skipped 2 previous similar messages [ 4817.298563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4821.814686] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4831.877631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.067693] Lustre: 178918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4832.389597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.206850] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4847.299978] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4854.207587] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4862.822874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4863.308473] Lustre: 180892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4869.926494] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4872.847404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4872.970450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4879.443879] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4888.553972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.555513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4888.579025] Lustre: Skipped 3 previous similar messages [ 4888.595576] LustreError: Skipped 2 previous similar messages [ 4888.600300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.616660] Lustre: Skipped 10 previous similar messages [ 4900.319104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4900.507196] Lustre: server umount lustre-OST0000 complete [ 4903.906206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4918.239630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4918.522860] Lustre: server umount lustre-MDT0000 complete [ 4919.276036] LustreError: 181356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.291061] LustreError: 181356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4921.803642] LustreError: 178908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063826 with bad export cookie 18389797529186882643 [ 4921.805487] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.822210] LustreError: 178908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4922.125058] Lustre: server umount lustre-MDT0001 complete [ 4931.468931] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 4934.029425] Key type lgssc unregistered [ 4934.355318] LNet: 182752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4934.368040] LNetError: 182752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4934.384606] LNet: Removed LNI 192.168.206.103@tcp [ 4935.174847] Key type .llcrypt unregistered [ 4935.177086] Key type ._llcrypt unregistered [ 4954.962759] Key type ._llcrypt registered [ 4954.966199] Key type .llcrypt registered [ 4955.067467] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 4956.013624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4956.032444] alg: No test for adler32 (adler32-zlib) [ 4957.076696] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 4957.301718] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 4959.015356] Key type lgssc registered [ 4960.006762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4967.177080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4967.190908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.638235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.595520] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4978.148367] LustreError: 184151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.191929] LustreError: 184151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4981.258528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4981.685618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4986.140294] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 4993.774669] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4998.785467] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5004.239435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.521870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.424586] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5014.778582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5018.203440] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5035.492933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5035.493883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.505646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5035.514915] Lustre: Skipped 1 previous similar message [ 5040.462866] Lustre: server umount lustre-OST0000 complete [ 5043.169437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5043.175720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.182922] Lustre: Skipped 1 previous similar message [ 5043.185727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5045.732404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.733034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5045.748425] Lustre: Skipped 1 previous similar message [ 5049.345296] Lustre: server umount lustre-MDT0000 complete [ 5050.851458] LustreError: 184150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.865460] LustreError: 184150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5052.293137] LustreError: 184132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063957 with bad export cookie 15983754698886790288 [ 5052.296995] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.302391] LustreError: 184132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5052.649613] Lustre: server umount lustre-MDT0001 complete [ 5061.325906] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 5063.870602] Key type lgssc unregistered [ 5064.137561] LNet: 187707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5064.147835] LNetError: 187707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5064.171413] LNet: Removed LNI 192.168.206.103@tcp [ 5064.959407] Key type .llcrypt unregistered [ 5064.962440] Key type ._llcrypt unregistered [ 5084.248952] Key type ._llcrypt registered [ 5084.251189] Key type .llcrypt registered [ 5084.342516] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 5085.303251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5085.402517] alg: No test for adler32 (adler32-zlib) [ 5086.533104] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 5086.813368] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 5088.495228] Key type lgssc registered [ 5089.766678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5097.549974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5097.581168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.076685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.925043] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5109.218352] LustreError: 189107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.233976] LustreError: 189107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5110.557735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.929517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5114.799398] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5121.854405] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5126.621643] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5132.032644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5132.287283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.411083] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5143.720501] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5144.563581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5154.607193] Lustre: lustre-OST0000: Not available for connect from 192.168.206.3@tcp (stopping) [ 5154.789971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.791904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5159.029908] Lustre: server umount lustre-OST0000 complete [ 5165.637485] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9aa0c7766800.ost_server_uuid 50 [ 5167.017928] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5182.434104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.446629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.471084] Lustre: Skipped 1 previous similar message [ 5182.482136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.496796] Lustre: Skipped 2 previous similar messages [ 5184.480144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.491621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.371717] Lustre: server umount lustre-MDT0000 complete [ 5189.605190] LustreError: 189102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.621283] LustreError: 189102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5189.708570] LustreError: 189088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064094 with bad export cookie 5561248614218582319 [ 5189.717713] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.725176] LustreError: 189088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5190.053553] Lustre: server umount lustre-MDT0001 complete [ 5197.970658] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:55:01 (1786064101) [ 5202.664649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.975456] LustreError: 193019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.050774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.957964] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5208.547809] LustreError: 193020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.594187] LustreError: 193020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5212.649379] LustreError: 193019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.358094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.818570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5219.270447] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5226.738227] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5231.778708] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5238.098646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5238.482748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.923695] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5247.746282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5250.950645] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5257.693739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5257.873539] Lustre: lustre-OST0001: new disk, initializing [ 5257.878055] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5257.884068] Lustre: 195863:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5257.964410] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5264.370745] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5265.487616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5265.508880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5265.611754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5272.623494] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5276.137890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.139023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5276.148470] Lustre: Skipped 1 previous similar message [ 5276.156012] Lustre: Skipped 2 previous similar messages [ 5281.071846] Lustre: server umount lustre-OST0000 complete [ 5281.258354] LustreError: 194840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.275731] LustreError: 194840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5286.865468] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5287.052939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5291.000733] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5291.202400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5298.982048] LustreError: 196295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.995096] LustreError: 196295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5306.856502] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.856996] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5306.873564] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5306.875241] LustreError: Skipped 1 previous similar message [ 5306.886295] Lustre: Skipped 2 previous similar messages [ 5310.979702] Lustre: server umount lustre-OST0001 complete [ 5317.093565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.110238] Lustre: Skipped 1 previous similar message [ 5317.118224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5317.131044] Lustre: Skipped 1 previous similar message [ 5329.375465] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.576786] Lustre: server umount lustre-MDT0000 complete [ 5331.426327] LustreError: 194858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.445934] LustreError: 194858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5333.238980] LustreError: 193000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064238 with bad export cookie 5561248614218583390 [ 5333.246448] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.259236] LustreError: 193000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5333.519884] Lustre: server umount lustre-MDT0001 complete [ 5353.214115] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:57:36 (1786064256) [ 5359.194085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.579755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5359.623860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5359.869167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.610546] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5370.751279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.875723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5370.921729] Lustre: 199305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5376.328521] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5383.436203] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5389.124888] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5394.745106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.910084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5399.660701] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5404.798331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5405.262945] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5410.773295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5410.829473] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5417.517752] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5424.610446] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5430.240158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5430.252656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.271874] Lustre: Skipped 1 previous similar message [ 5430.289536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.298977] Lustre: Skipped 4 previous similar messages [ 5433.315796] Lustre: server umount lustre-OST0000 complete [ 5435.680537] LustreError: 201155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.720586] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5454.941955] Lustre: server umount lustre-OST0001 complete [ 5466.593977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.601339] Lustre: Skipped 8 previous similar messages [ 5473.247298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5473.416548] Lustre: server umount lustre-MDT0000 complete [ 5476.369964] LustreError: 199297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064381 with bad export cookie 5561248614218584790 [ 5476.374148] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.379975] LustreError: 199297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5476.663989] Lustre: server umount lustre-MDT0001 complete [ 5496.819844] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:00:00 (1786064400) [ 5511.862456] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 5520.304458] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 5552.223686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.392585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5552.417738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5552.533586] Lustre: lustre-MDT0000: new disk, initializing [ 5552.600292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.608307] Lustre: Skipped 3 previous similar messages [ 5552.620381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5556.914187] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5566.847291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.029550] Lustre: 207398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5567.084189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5567.095976] Lustre: Skipped 1 previous similar message [ 5567.185636] Lustre: lustre-MDT0001: new disk, initializing [ 5567.360470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5567.374757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5571.346557] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5579.855845] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5584.826359] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5591.216519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.448374] Lustre: lustre-OST0000: new disk, initializing [ 5591.451910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5591.462325] Lustre: 209376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5597.044434] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5598.791958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5598.809156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5598.904666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5605.602349] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5610.197725] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5610.409549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5614.622122] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5614.816570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5619.683584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5619.686562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.690280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5619.700917] Lustre: Skipped 6 previous similar messages [ 5619.711216] Lustre: Skipped 3 previous similar messages [ 5622.600581] Lustre: server umount lustre-OST0000 complete [ 5632.204412] Lustre: server umount lustre-MDT0000 complete [ 5635.046989] LustreError: 209800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.071427] LustreError: 209800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5635.521981] LustreError: 209375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064540 with bad export cookie 5561248614218586351 [ 5635.522469] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.529797] LustreError: 209375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5635.881852] Lustre: server umount lustre-MDT0001 complete [ 5644.234857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.645861] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5656.868285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.562578] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5668.901844] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5675.043417] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5681.523065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5681.881033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.888688] Lustre: Skipped 4 previous similar messages [ 5688.156911] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5694.646658] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5698.662895] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5698.859568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5703.301184] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5703.513215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5708.770848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.776972] LustreError: Skipped 2 previous similar messages [ 5708.781884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.794666] Lustre: Skipped 2 previous similar messages [ 5711.680787] Lustre: server umount lustre-OST0000 complete [ 5717.366347] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5721.864851] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5750.567154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5757.279603] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5759.153915] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.186301] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5765.062340] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5769.763286] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5774.249242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.256025] Lustre: Skipped 6 previous similar messages [ 5779.480836] Lustre: server umount lustre-OST0000 complete [ 5789.513982] Lustre: server umount lustre-MDT0000 complete [ 5793.372734] LustreError: 211575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064698 with bad export cookie 5561248614218587240 [ 5793.381355] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5793.388087] LustreError: 211575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5794.023178] Lustre: server umount lustre-MDT0001 complete [ 5803.891977] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:05:07 (1786064707) [ 5810.600772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.044081] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5825.409939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.303795] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5837.966223] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5843.831143] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5851.007361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.703148] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5865.798398] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5871.096443] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5871.333760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5876.575922] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5876.877503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5884.929532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5885.215856] Lustre: lustre-OST0001: new disk, initializing [ 5885.224236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5885.237412] Lustre: 220929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5891.823384] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5894.849223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5894.867222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5894.988781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5902.260611] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5907.326927] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5907.656236] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5912.449864] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5912.666152] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5915.621542] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.637727] Lustre: Skipped 4 previous similar messages [ 5915.647709] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.655471] LustreError: Skipped 1 previous similar message [ 5920.533515] Lustre: server umount lustre-OST0001 complete [ 5920.747416] LustreError: 219383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.767164] LustreError: 219383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5927.061905] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5927.264386] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5932.051311] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5932.300945] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5961.409734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.623637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.632401] Lustre: Skipped 5 previous similar messages [ 5966.829585] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5966.857225] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5966.858091] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5966.880202] Lustre: Skipped 1 previous similar message [ 5968.488961] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 5977.703691] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5980.166662] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5987.351127] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.373464] LustreError: Skipped 1 previous similar message [ 5988.798267] Lustre: server umount lustre-OST0001 complete [ 6011.400344] Lustre: server umount lustre-OST0000 complete [ 6029.281248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6029.427681] Lustre: server umount lustre-MDT0000 complete [ 6033.423494] LustreError: 217541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064938 with bad export cookie 5561248614218588199 [ 6033.435953] LustreError: 217541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6033.438579] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6039.854827] Lustre: server umount lustre-MDT0001 complete [ 6062.600880] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:09:26 (1786064966) [ 6069.070425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.362498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6069.406058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6073.836038] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6080.875994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.949457] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6080.977327] Lustre: 226173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6085.831417] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6093.122588] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.088974] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6106.435971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.674582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6112.798770] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6119.451787] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.456558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6128.514979] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6134.746527] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6141.974819] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.710617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6147.896697] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6148.131931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6152.591465] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6152.840280] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6158.747691] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa0d2273000.ost_server_uuid 50 [ 6160.167149] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aa0d2273000.ost_server_uuid in IDLE state after 0 sec [ 6161.368159] Lustre: Permanently deactivating lustre-OST0001 [ 6161.393168] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6164.806490] Lustre: Permanently reactivating lustre-OST0001 [ 6170.093884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6170.094263] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6170.101908] Lustre: Skipped 18 previous similar messages [ 6170.114458] LustreError: Skipped 3 previous similar messages [ 6175.822967] Lustre: server umount lustre-OST0001 complete [ 6180.331449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.344029] Lustre: Skipped 9 previous similar messages [ 6185.373280] Lustre: server umount lustre-OST0000 complete [ 6194.772590] Lustre: server umount lustre-MDT0000 complete [ 6197.533840] LustreError: 226167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065102 with bad export cookie 5561248614218589487 [ 6197.540414] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6197.543014] LustreError: 226167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6197.839614] Lustre: server umount lustre-MDT0001 complete [ 6216.952610] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:12:00 (1786065120) [ 6224.219396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.508436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6224.550686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6224.556316] Lustre: Skipped 5 previous similar messages [ 6229.143550] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6236.389619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.469287] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6236.493371] Lustre: 232869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6240.571879] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6247.531388] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6253.319589] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6259.241420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.394479] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6259.411354] Lustre: Permanently deactivating lustre-OST0000 [ 6259.420835] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6264.785739] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6265.856223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6265.927468] LustreError: 235167:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6271.047607] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6276.871682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.960922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6282.873167] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6283.353290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6289.320881] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6290.829835] Lustre: Permanently reactivating lustre-OST0000 [ 6290.845702] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6290.849297] Lustre: Skipped 2 previous similar messages [ 6297.684298] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6297.685305] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 0000000023cc5a8a, cur 1786065202 deadline 1786065200 last 1786065170 [ 6297.690315] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.696779] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6297.735709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6306.839819] Lustre: server umount lustre-OST0001 complete [ 6317.355641] Lustre: server umount lustre-OST0000 complete [ 6327.657334] Lustre: server umount lustre-MDT0000 complete [ 6331.021238] LustreError: 233610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065235 with bad export cookie 5561248614218591167 [ 6331.025291] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.038871] LustreError: 233610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6331.348954] Lustre: server umount lustre-MDT0001 complete [ 6341.926874] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 6344.590872] Key type lgssc unregistered [ 6344.938197] LNet: 237748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6344.946628] LNetError: 237748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6344.975791] LNet: Removed LNI 192.168.206.103@tcp [ 6345.786887] Key type .llcrypt unregistered [ 6345.788414] Key type ._llcrypt unregistered [ 6366.609663] Key type ._llcrypt registered [ 6366.614921] Key type .llcrypt registered [ 6366.732855] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6367.766956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6367.883063] alg: No test for adler32 (adler32-zlib) [ 6368.950837] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6369.201196] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 6370.895443] Key type lgssc registered [ 6371.851717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6380.558083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6380.578693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.131157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.415444] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6392.291115] LustreError: 239217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.318066] LustreError: 239217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6394.893718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.077277] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6395.107028] Lustre: 239206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6395.117350] Lustre: Permanently deactivating lustre-MDT0001 [ 6395.131252] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6395.506621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6396.226259] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6396.247256] LustreError: 239247:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6396.270804] LustreError: 239247:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6396.294332] LustreError: 239247:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 6400.389117] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6408.104689] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6412.860166] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6420.133018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.437764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.483984] LustreError: 241055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.503619] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6426.310642] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6426.600142] LustreError: 241055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6427.643267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6432.739703] LustreError: 241057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.783804] LustreError: 241057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6433.163720] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6437.861065] LustreError: 241055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.872807] LustreError: 241055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6439.263216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.520126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6444.552431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6444.557624] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6445.347303] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6452.773472] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6456.777599] Lustre: Permanently reactivating lustre-MDT0001 [ 6456.791981] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6456.798835] Lustre: Skipped 2 previous similar messages [ 6463.280071] Lustre: lustre-MDT0001: Client 10b121c9-e016-4abf-9c5b-8a24e1e08e7a (at 192.168.206.3@tcp) reconnecting [ 6468.278584] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.624906] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6474.439858] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6474.605260] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6478.452168] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6478.623552] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6480.843710] Lustre: Permanently deactivating lustre-MDT0001 [ 6480.867662] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6480.875513] Lustre: Skipped 2 previous similar messages [ 6488.147469] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6495.714349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.724814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.738435] Lustre: Skipped 3 previous similar messages [ 6495.747508] Lustre: Skipped 2 previous similar messages [ 6499.511443] Lustre: server umount lustre-MDT0000 complete [ 6500.835323] LustreError: 239217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.857473] LustreError: 239217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6503.293608] LustreError: 239197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065408 with bad export cookie 2135587700909492475 [ 6503.294167] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.324726] LustreError: 239197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6503.692369] Lustre: server umount lustre-MDT0001 complete [ 6509.206432] Lustre: server umount lustre-OST0000 complete [ 6519.590185] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:17:03 (1786065423) [ 6529.316414] Lustre: server umount lustre-OST0001 complete [ 6536.123056] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 6545.563749] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6578.143271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.397455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6578.408834] Lustre: Skipped 2 previous similar messages [ 6578.441976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6578.533826] Lustre: lustre-MDT0000: new disk, initializing [ 6578.629120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6578.655132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6583.276550] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6593.527133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.653312] Lustre: 247472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6593.720461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6593.725647] Lustre: Skipped 1 previous similar message [ 6593.819643] Lustre: lustre-MDT0001: new disk, initializing [ 6593.910989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6593.948621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6593.975475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6598.469131] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6607.170134] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6611.892335] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6618.541818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.839537] Lustre: lustre-OST0000: new disk, initializing [ 6618.843577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6618.849564] Lustre: 249441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6618.979714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.981780] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6628.431416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6628.440824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6628.570854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6635.557201] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6637.584156] LustreError: 247479:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6647.078550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.294272] Lustre: lustre-OST0001: new disk, initializing [ 6647.301442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6647.306412] Lustre: 250554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6647.594254] LustreError: 247479:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6647.618548] LustreError: 250398:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6653.629388] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6654.076382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6654.090221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6654.187738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6657.639151] LustreError: 250398:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6662.053123] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6663.479199] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6669.805340] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.806707] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.809806] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6669.809814] Lustre: Skipped 1 previous similar message [ 6669.819156] Lustre: Skipped 1 previous similar message [ 6669.830321] LustreError: Skipped 1 previous similar message [ 6671.400888] Lustre: server umount lustre-OST0001 complete [ 6674.920748] LustreError: 249426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.942104] LustreError: 249426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6690.277059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.280739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.282148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.282154] Lustre: Skipped 1 previous similar message [ 6690.292789] LustreError: Skipped 1 previous similar message [ 6690.316956] Lustre: Skipped 1 previous similar message [ 6694.923411] Lustre: server umount lustre-OST0000 complete [ 6700.513067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.528210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.537613] Lustre: Skipped 1 previous similar message [ 6704.394882] Lustre: server umount lustre-MDT0000 complete [ 6707.467181] LustreError: 248267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065612 with bad export cookie 2135587700909495289 [ 6707.475907] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.479109] LustreError: 248267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6714.141760] Lustre: server umount lustre-MDT0001 complete [ 6725.625902] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 6728.897724] Key type lgssc unregistered [ 6729.162251] LNet: 252592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6729.175274] LNetError: 252592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6730.219340] LNet: Removed LNI 192.168.206.103@tcp [ 6731.229432] Key type .llcrypt unregistered [ 6731.234926] Key type ._llcrypt unregistered [ 6770.797534] Key type ._llcrypt registered [ 6770.802320] Key type .llcrypt registered [ 6770.948880] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 6772.159709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6772.219044] alg: No test for adler32 (adler32-zlib) [ 6773.272485] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6773.487070] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 6775.159222] Key type lgssc registered [ 6776.204033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6782.975305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6782.990918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.205465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6784.237408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6784.417955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.221380] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6795.579359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.640522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6795.703367] Lustre: 254805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6796.021626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6799.881923] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6806.779869] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6811.650663] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6817.114326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.312631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6817.514336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.342969] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6826.633627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6830.225021] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6882.789625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6882.797510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.815112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.301093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.301627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6883.318901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.325255] Lustre: Skipped 1 previous similar message [ 6886.431513] Lustre: server umount lustre-OST0000 complete [ 6890.397355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6903.466953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.804375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.904418] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6908.934101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6908.944209] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6910.755446] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 6917.541279] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.977303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.982882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6955.002739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6955.025183] Lustre: Skipped 1 previous similar message [ 6960.107989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6960.120281] Lustre: Skipped 1 previous similar message [ 6961.014552] Lustre: server umount lustre-OST0000 complete [ 6965.216521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6965.231278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.242622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6965.259967] Lustre: Skipped 2 previous similar messages [ 6965.275101] Lustre: Skipped 1 previous similar message [ 6970.699063] Lustre: server umount lustre-MDT0000 complete [ 6974.294370] LustreError: 254796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065879 with bad export cookie 5483133265112216911 [ 6974.295708] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.300418] LustreError: 254796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6974.934200] Lustre: server umount lustre-MDT0001 complete [ 6985.000570] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 6987.644937] Key type lgssc unregistered [ 6987.858078] LNet: 260226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.866752] LNetError: 260226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6987.877312] LNet: Removed LNI 192.168.206.103@tcp [ 6988.709909] Key type .llcrypt unregistered [ 6988.714713] Key type ._llcrypt unregistered [ 7016.742332] Key type ._llcrypt registered [ 7016.745887] Key type .llcrypt registered [ 7016.898792] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7018.459746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7018.527282] alg: No test for adler32 (adler32-zlib) [ 7019.598757] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7019.910059] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7021.561478] Key type lgssc registered [ 7022.777286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7030.982637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7031.010768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.492350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.232298] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7043.043234] LustreError: 261645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.064345] LustreError: 261645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7043.230855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.592740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7048.589645] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7056.935833] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7063.287306] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7070.023757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.348192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.083863] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7078.576706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7082.713179] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.721051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7114.724693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.736600] LustreError: Skipped 1 previous similar message [ 7114.745678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7114.748567] Lustre: Skipped 1 previous similar message [ 7114.759059] Lustre: Skipped 1 previous similar message [ 7117.529448] Lustre: server umount lustre-OST0000 complete [ 7124.967681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.979148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.990309] Lustre: Skipped 1 previous similar message [ 7130.094238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7130.100663] Lustre: Skipped 1 previous similar message [ 7134.178169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.199159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7135.451638] Lustre: server umount lustre-MDT0000 complete [ 7139.297972] LustreError: 261644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.336870] LustreError: 261644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7139.468601] LustreError: 261625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066044 with bad export cookie 6716203411612700084 [ 7139.477672] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.488107] LustreError: 261625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.880217] Lustre: server umount lustre-MDT0001 complete [ 7152.850066] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7156.501173] Key type lgssc unregistered [ 7157.044113] LNet: 265955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7157.061808] LNetError: 265955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.089638] LNet: Removed LNI 192.168.206.103@tcp [ 7158.167396] Key type .llcrypt unregistered [ 7158.168709] Key type ._llcrypt unregistered [ 7174.594842] Key type ._llcrypt registered [ 7174.598801] Key type .llcrypt registered [ 7174.699967] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7175.793450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7175.943213] alg: No test for adler32 (adler32-zlib) [ 7177.077899] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7177.474400] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7179.192345] Key type lgssc registered [ 7180.479152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7188.472787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7188.494062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.002484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.653828] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7199.201874] LustreError: 267191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7199.217089] LustreError: 267191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7203.626908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.070652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7208.679389] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7216.851310] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7222.291455] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7228.985739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.422554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.562176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7236.044498] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7242.881126] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.108392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.122623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.126918] Lustre: Skipped 1 previous similar message [ 7259.139929] Lustre: Skipped 1 previous similar message [ 7263.505385] Lustre: server umount lustre-OST0000 complete [ 7269.345666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.352308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7269.370363] Lustre: Skipped 1 previous similar message [ 7269.390099] Lustre: Skipped 1 previous similar message [ 7274.465476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.480611] Lustre: Skipped 1 previous similar message [ 7274.986323] Lustre: server umount lustre-MDT0000 complete [ 7278.916933] LustreError: 267173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066183 with bad export cookie 14755930741002587918 [ 7278.930151] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.266826] Lustre: server umount lustre-MDT0001 complete [ 7290.903605] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7293.485535] Key type lgssc unregistered [ 7293.756833] LNet: 270871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.762341] LNetError: 270871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.775338] LNet: Removed LNI 192.168.206.103@tcp [ 7294.588283] Key type .llcrypt unregistered [ 7294.590285] Key type ._llcrypt unregistered [ 7315.781238] Key type ._llcrypt registered [ 7315.782786] Key type .llcrypt registered [ 7315.861440] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7316.792857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.864798] alg: No test for adler32 (adler32-zlib) [ 7317.919267] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7318.144321] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7319.783154] Key type lgssc registered [ 7320.952983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7327.461286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7327.497284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.059837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7332.829921] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7338.480897] LustreError: 272287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.514165] LustreError: 272287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7341.047831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.505578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7345.565151] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7351.870749] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.180927] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7361.332387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.840716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.795369] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7373.117525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7374.583739] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7398.883558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.893079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7398.912075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.919527] Lustre: Skipped 1 previous similar message [ 7404.003697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7404.006639] Lustre: Skipped 1 previous similar message [ 7404.494832] Lustre: server umount lustre-OST0000 complete [ 7412.193346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.196202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.201885] Lustre: Skipped 2 previous similar messages [ 7412.214419] Lustre: Skipped 1 previous similar message [ 7414.785155] Lustre: server umount lustre-MDT0000 complete [ 7417.323683] LustreError: 272288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.348701] LustreError: 272288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7418.399559] LustreError: 272268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066323 with bad export cookie 17933710396362024423 [ 7418.405590] LustreError: 272268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7418.412268] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7418.719774] Lustre: server umount lustre-MDT0001 complete [ 7428.962746] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7431.548520] Key type lgssc unregistered [ 7431.819248] LNet: 276597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.835647] LNetError: 276597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.854133] LNet: Removed LNI 192.168.206.103@tcp [ 7432.996207] Key type .llcrypt unregistered [ 7432.998390] Key type ._llcrypt unregistered [ 7450.130258] Key type ._llcrypt registered [ 7450.135300] Key type .llcrypt registered [ 7450.305569] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7451.585388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7451.890338] alg: No test for adler32 (adler32-zlib) [ 7453.028903] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7453.461274] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7455.239161] Key type lgssc registered [ 7456.510984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7463.608292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7463.621475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.208624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.609185] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7475.681341] LustreError: 277837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.722115] LustreError: 277837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7476.238578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.580227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7480.337329] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7486.627637] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.407427] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.742323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7498.198544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.738880] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7511.554646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7513.112857] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.881633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7526.887254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.895720] LustreError: Skipped 1 previous similar message [ 7526.902308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7526.922268] Lustre: Skipped 1 previous similar message [ 7526.943689] Lustre: Skipped 1 previous similar message [ 7528.808961] Lustre: server umount lustre-OST0000 complete [ 7532.004730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.010911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7532.023846] Lustre: Skipped 1 previous similar message [ 7532.032871] Lustre: Skipped 1 previous similar message [ 7537.121264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.137845] Lustre: Skipped 1 previous similar message [ 7542.240402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.335145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7546.339271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.356807] Lustre: Skipped 2 previous similar messages [ 7546.482689] Lustre: server umount lustre-MDT0000 complete [ 7549.918842] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066454 with bad export cookie 6805386343330467825 [ 7549.923650] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.930762] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.547984] Lustre: server umount lustre-MDT0001 complete [ 7565.768857] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7570.130717] Key type lgssc unregistered [ 7570.579968] LNet: 281542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7570.590399] LNetError: 281542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7570.617341] LNet: Removed LNI 192.168.206.103@tcp [ 7571.898342] Key type .llcrypt unregistered [ 7571.900251] Key type ._llcrypt unregistered [ 7580.867447] Key type ._llcrypt registered [ 7580.869957] Key type .llcrypt registered [ 7580.967395] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7595.231333] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 7610.753405] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7611.615503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7611.733753] alg: No test for adler32 (adler32-zlib) [ 7613.028605] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7613.340406] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7615.095186] Key type lgssc registered [ 7616.174666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7648.808310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7648.825091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.991310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7650.019997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7650.103741] Lustre: lustre-MDT0000: new disk, initializing [ 7650.184102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.206439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7653.809355] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7663.901076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.084572] Lustre: 285091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7664.156694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7664.161807] Lustre: Skipped 1 previous similar message [ 7664.282279] Lustre: lustre-MDT0001: new disk, initializing [ 7664.450840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.532963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7664.550741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7669.639285] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7680.309574] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7685.827784] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7691.693124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.044573] Lustre: lustre-OST0000: new disk, initializing [ 7692.047895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7692.052561] Lustre: 287059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7692.102336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7698.030942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7698.042906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7698.065299] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7698.176763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7706.702404] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7711.293500] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7711.560173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7716.187163] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7716.452651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7718.880436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7718.886306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.896792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7724.005437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7724.021850] Lustre: Skipped 2 previous similar messages [ 7724.574240] Lustre: server umount lustre-OST0000 complete [ 7729.122591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7729.133148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7729.139835] Lustre: Skipped 2 previous similar messages [ 7729.153226] Lustre: Skipped 1 previous similar message [ 7734.253241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.270298] Lustre: Skipped 1 previous similar message [ 7734.481127] Lustre: server umount lustre-MDT0000 complete [ 7738.204478] LustreError: 285083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066643 with bad export cookie 18270652113830153068 [ 7738.207472] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.220725] LustreError: 285083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7738.652790] Lustre: server umount lustre-MDT0001 complete [ 7748.379530] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:37:31 (1786066651) [ 7755.818361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.277421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.913320] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7761.380319] LustreError: 289474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.407406] LustreError: 289474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7766.513189] LustreError: 289473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.257049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7769.488297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7774.064873] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7781.519472] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.328229] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.914466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.102497] LustreError: 291296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.349383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7799.424175] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7805.616863] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7815.136073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7815.139923] LustreError: Skipped 1 previous similar message [ 7815.144497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.156566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.192743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.193534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.828799] Lustre: server umount lustre-OST0000 complete [ 7825.904066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7825.919206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.927120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.929711] Lustre: Skipped 2 previous similar messages [ 7831.672275] Lustre: server umount lustre-MDT0000 complete [ 7832.547789] LustreError: 289470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.571350] LustreError: 289470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7835.592585] LustreError: 290208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066740 with bad export cookie 18270652113830153950 [ 7835.594283] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7835.606596] LustreError: 290208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7836.047174] Lustre: server umount lustre-MDT0001 complete [ 7851.507412] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 7855.462417] Key type lgssc unregistered [ 7855.726511] LNet: 293012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7855.734337] LNetError: 293012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7855.744745] LNet: Removed LNI 192.168.206.103@tcp [ 7856.503493] Key type .llcrypt unregistered [ 7856.505877] Key type ._llcrypt unregistered [ 7885.649816] Key type ._llcrypt registered [ 7885.651274] Key type .llcrypt registered [ 7885.783493] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 7902.043943] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 7902.954639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7903.096531] alg: No test for adler32 (adler32-zlib) [ 7904.167698] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7904.526051] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 7906.175161] Key type lgssc registered [ 7907.344276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7950.877359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7950.919733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.225174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7952.282188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7952.380920] Lustre: lustre-MDT0000: new disk, initializing [ 7952.490587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.525194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7956.040094] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7965.212585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.454795] Lustre: 296839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7965.685721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7965.701648] Lustre: Skipped 1 previous similar message [ 7965.776655] Lustre: lustre-MDT0001: new disk, initializing [ 7965.974569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7966.052249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7966.075169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7972.000129] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 7983.185130] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7988.306500] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7994.237025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7994.440535] Lustre: lustre-OST2710: new disk, initializing [ 7994.443778] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7994.448736] Lustre: 298805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7994.501121] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8000.905879] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8002.646559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8002.679957] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8002.810225] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8010.480462] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.044267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8017.316425] Lustre: lustre-OST03e8: new disk, initializing [ 8017.320808] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8017.327419] Lustre: 299869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8017.425582] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8022.757655] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8023.099853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8023.111630] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8023.186411] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8031.005208] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8038.862812] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8039.120796] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8047.231509] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8047.593700] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8052.978847] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8053.189678] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8058.254023] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8058.417244] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8063.456742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8063.470487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.484371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.323971] Lustre: server umount lustre-MDT0000 complete [ 8064.493148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.506506] Lustre: Skipped 1 previous similar message [ 8067.453325] LustreError: 299868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066972 with bad export cookie 5139859834162098261 [ 8067.472178] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8067.941933] Lustre: server umount lustre-MDT0001 complete [ 8081.916495] Lustre: server umount lustre-OST2710 complete [ 8096.201199] Lustre: server umount lustre-OST03e8 complete [ 8113.868837] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 8122.603885] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8156.734661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8157.093688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8157.149449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8157.220702] Lustre: lustre-MDT0000: new disk, initializing [ 8157.291366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.304644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8161.946291] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8171.763838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.958965] Lustre: 305118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8172.058776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8172.070677] Lustre: Skipped 1 previous similar message [ 8172.134439] Lustre: lustre-MDT0001: new disk, initializing [ 8172.299826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.338100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8172.348508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8176.835231] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8185.796246] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8192.112656] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8198.767421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.015661] Lustre: lustre-OST0000: new disk, initializing [ 8199.019604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8199.032963] Lustre: 307085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8199.149862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.772777] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8206.407156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8206.414469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8206.501802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8212.136888] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8216.090157] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8216.316607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8220.381901] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8220.638909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8227.301433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.302586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8227.317073] Lustre: Skipped 1 previous similar message [ 8227.330785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8227.332086] LustreError: Skipped 1 previous similar message [ 8228.429160] Lustre: server umount lustre-OST0000 complete [ 8232.418984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.432414] Lustre: Skipped 1 previous similar message [ 8232.439629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8232.445320] Lustre: Skipped 2 previous similar messages [ 8237.536764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.548104] Lustre: Skipped 1 previous similar message [ 8237.997752] Lustre: server umount lustre-MDT0000 complete [ 8241.206452] LustreError: 305112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067146 with bad export cookie 5139859834162099941 [ 8241.208369] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8241.213791] LustreError: 305112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8241.603782] Lustre: server umount lustre-MDT0001 complete [ 8248.697664] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:45:52 (1786067152) [ 8249.862408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8251.289646] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:45:55 (1786067155) [ 8254.207392] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8279.877980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.166551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8280.225412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8280.520985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.765325] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8291.978028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.094672] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8292.133548] Lustre: 310942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8295.689669] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8315.635269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8315.893824] Lustre: Denying initial registration attempt from nid 192.168.206.103@tcp, specified as failover [ 8315.912496] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8315.934646] LustreError: 312394:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8315.942402] LustreError: 312394:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8315.948469] LustreError: 312394:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8316.029328] Lustre: server umount lustre-OST0000 complete [ 8316.033105] LustreError: 312394:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8323.042276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8323.051306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.063530] Lustre: Skipped 1 previous similar message [ 8323.066308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8327.551105] Lustre: server umount lustre-MDT0000 complete [ 8329.706128] LustreError: 310954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.718689] LustreError: 310954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8331.693330] LustreError: 310935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067236 with bad export cookie 5139859834162100823 [ 8331.703642] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.720811] LustreError: 310935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8332.334077] Lustre: server umount lustre-MDT0001 complete [ 8344.417802] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 8347.499327] Key type lgssc unregistered [ 8347.700354] LNet: 313517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8347.711225] LNetError: 313517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8348.773695] LNet: Removed LNI 192.168.206.103@tcp [ 8349.504342] Key type .llcrypt unregistered [ 8349.506146] Key type ._llcrypt unregistered [ 8360.278967] Key type ._llcrypt registered [ 8360.286247] Key type .llcrypt registered [ 8360.380632] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8361.329667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8361.452029] alg: No test for adler32 (adler32-zlib) [ 8362.461854] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 8362.675385] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 8364.375251] Key type lgssc registered [ 8365.346538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8410.030547] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8416.172945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8416.194455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.359399] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8417.385917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8417.503271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.979843] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8429.669793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.738876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8429.756782] Lustre: 316283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8429.964829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8434.010211] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8440.240644] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8445.527567] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8464.932332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8465.110975] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8465.385078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.939239] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8478.871814] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8483.301175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.326923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.913699] Lustre: server umount lustre-OST0000 complete [ 8491.489440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8491.496973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.518464] Lustre: Skipped 1 previous similar message [ 8491.533085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8491.542097] Lustre: Skipped 1 previous similar message [ 8493.538197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.539540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.365484] Lustre: server umount lustre-MDT0000 complete [ 8499.534943] LustreError: 317024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067404 with bad export cookie 10822501875503210993 [ 8499.536402] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8499.542896] LustreError: 317024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8499.849425] Lustre: server umount lustre-MDT0001 complete [ 8509.311780] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 8512.088458] Key type lgssc unregistered [ 8512.612853] LNet: 319856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8512.634383] LNetError: 319856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8512.666322] LNet: Removed LNI 192.168.206.103@tcp [ 8513.817468] Key type .llcrypt unregistered [ 8513.820315] Key type ._llcrypt unregistered [ 8541.962454] Key type ._llcrypt registered [ 8541.969489] Key type .llcrypt registered [ 8542.126573] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8544.114861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8544.395182] alg: No test for adler32 (adler32-zlib) [ 8545.782705] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 8546.104816] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 8547.943925] Key type lgssc registered [ 8549.112893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8559.174105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8559.198280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.637625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.318772] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8570.850259] LustreError: 321276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8570.861967] LustreError: 321276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8572.347802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.652460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8576.404956] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8583.272987] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8588.308880] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8596.474729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8596.973714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.805135] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8611.306811] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8618.298511] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [ 8618.975942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8618.981077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8618.991840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.513677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.517464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.531651] Lustre: Skipped 1 previous similar message [ 8620.552347] Lustre: Skipped 1 previous similar message [ 8623.389633] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [ 8623.398442] Lustre: Skipped 1 previous similar message [ 8624.347617] Lustre: server umount lustre-MDT0000 complete [ 8625.635954] LustreError: 322005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8625.653930] LustreError: 322005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8627.122115] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067532 with bad export cookie 12900584076837896237 [ 8627.125063] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.128569] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8628.515198] LustreError: 321270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.538776] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.3@tcp (stopping) [ 8630.754307] LustreError: 321272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.758851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.448198] Lustre: server umount lustre-MDT0001 complete [ 8638.821025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8646.196320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.521928] LustreError: 323108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.542532] LustreError: 323108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8652.625156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.686068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8657.159889] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8662.888278] LustreError: 324865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.891063] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8662.919415] LustreError: 324865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8665.554515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.935371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8670.139988] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8671.226604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8676.479548] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.846828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8682.588837] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8683.848768] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8691.681304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.684166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.698300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.717874] Lustre: Skipped 1 previous similar message [ 8691.730053] Lustre: Skipped 3 previous similar messages [ 8702.943139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8703.060494] Lustre: server umount lustre-OST0000 complete [ 8712.160222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.165885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8712.173934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.173943] Lustre: Skipped 4 previous similar messages [ 8712.179692] Lustre: Skipped 1 previous similar message [ 8713.774642] Lustre: server umount lustre-MDT0000 complete [ 8716.907795] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067621 with bad export cookie 12900584076837903426 [ 8716.924440] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8716.927311] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8717.282776] Lustre: server umount lustre-MDT0001 complete [ 8727.597590] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 8730.502802] Key type lgssc unregistered [ 8730.782361] LNet: 327544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8730.795247] LNetError: 327544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8730.814196] LNet: Removed LNI 192.168.206.103@tcp [ 8731.673703] Key type .llcrypt unregistered [ 8731.677235] Key type ._llcrypt unregistered [ 8753.891577] Key type ._llcrypt registered [ 8753.894875] Key type .llcrypt registered [ 8753.995892] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 8754.736438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8754.785941] alg: No test for adler32 (adler32-zlib) [ 8755.746594] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 8755.952867] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 8757.615168] Key type lgssc registered [ 8758.623562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8765.077959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8765.105175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8766.415090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.135720] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8776.680086] LustreError: 328955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.702305] LustreError: 328955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8777.505519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.897107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.444248] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8789.676789] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.436933] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8798.688206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8798.697103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8798.705886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8801.761741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8801.762766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.850477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8802.855488] Lustre: Skipped 2 previous similar messages [ 8804.344175] Lustre: server umount lustre-MDT0000 complete [ 8807.162150] LustreError: 328933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067712 with bad export cookie 6170954430195710713 [ 8807.167416] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.176043] LustreError: 328933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8807.459699] Lustre: server umount lustre-MDT0001 complete [ 8817.951279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8818.199880] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8818.226603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8818.339555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.490579] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8828.706799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.809183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8828.863303] Lustre: 331541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8829.223338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8833.386703] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8840.913979] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8847.760631] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8857.326052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8857.474678] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8857.479991] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8857.485243] LustreError: 333423:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 8857.488699] LustreError: 333423:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8857.491944] LustreError: 333423:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8857.531440] Lustre: server umount lustre-OST0000 complete [ 8857.534520] LustreError: 333423:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8866.560599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.953769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8867.347354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.488894] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8876.796773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8885.655841] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8897.497867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.621157] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8897.720369] Lustre: lustre-OST0001: new disk, initializing [ 8897.724237] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8897.730423] Lustre: 334968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8897.815220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8904.294049] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 8906.804074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8906.811510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8906.876246] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8911.334265] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8916.965237] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.966639] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8916.971035] Lustre: Skipped 1 previous similar message [ 8916.983839] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8920.437262] Lustre: server umount lustre-OST0001 complete [ 8922.088695] LustreError: 334339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.109389] LustreError: 334339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8927.204143] LustreError: 333890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.209361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.214707] LustreError: 333890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8927.236868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.243423] Lustre: Skipped 2 previous similar messages [ 8929.591689] Lustre: server umount lustre-OST0000 complete [ 8936.933077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8936.949603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.980445] Lustre: Skipped 1 previous similar message [ 8936.991605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8938.971504] Lustre: server umount lustre-MDT0000 complete [ 8942.430215] LustreError: 331533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786067847 with bad export cookie 6170954430195711266 [ 8942.438550] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8942.444917] LustreError: 331533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8942.826936] Lustre: server umount lustre-MDT0001 complete [ 8952.862766] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 8955.884902] Key type lgssc unregistered [ 8956.168953] LNet: 336926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8956.173508] LNetError: 336926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8956.185140] LNet: Removed LNI 192.168.206.103@tcp [ 8956.893526] Key type .llcrypt unregistered [ 8956.896398] Key type ._llcrypt unregistered [ 9035.038170] Key type ._llcrypt registered [ 9035.040413] Key type .llcrypt registered [ 9035.137523] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 9050.547486] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9051.946854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9051.986996] alg: No test for adler32 (adler32-zlib) [ 9053.078335] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 9053.379380] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 9055.151137] Key type lgssc registered [ 9056.270373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9092.166666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9092.221969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.489656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.538326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9093.603598] Lustre: lustre-MDT0000: new disk, initializing [ 9093.699293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.718675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9098.210647] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9107.519968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9107.697761] Lustre: 341862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9107.754216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9107.765102] Lustre: Skipped 1 previous similar message [ 9107.869238] Lustre: lustre-MDT0001: new disk, initializing [ 9108.023091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9108.086886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9108.101520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9112.776589] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9123.782047] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9129.376663] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9136.107720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.367680] Lustre: lustre-OST0000: new disk, initializing [ 9136.378823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9136.389763] Lustre: 343834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9136.525647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.735089] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9143.885425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9143.899353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9143.999828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9150.293718] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9154.702782] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9154.944392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9160.937668] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9161.097466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9164.777821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9164.791635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.816655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9169.301975] Lustre: server umount lustre-OST0000 complete [ 9174.496816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.518227] LustreError: Skipped 1 previous similar message [ 9174.534061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.543422] Lustre: Skipped 1 previous similar message [ 9174.546614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.549037] Lustre: Skipped 1 previous similar message [ 9179.356456] Lustre: server umount lustre-MDT0000 complete [ 9183.389571] LustreError: 341855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068088 with bad export cookie 2987161779094533946 [ 9183.409633] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.958345] Lustre: server umount lustre-MDT0001 complete [ 9193.090599] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:01:36 (1786068096) [ 9203.269378] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:01:45 (1786068105) [ 9214.764320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.801208] LustreError: 346600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.843292] LustreError: 346600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9216.028477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9221.099064] LustreError: 346601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9221.412335] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9225.196224] LustreError: 346600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9228.975415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.279749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.294986] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9239.930815] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9245.313835] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9253.084855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9253.610724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.391336] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9267.922752] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9273.992885] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9275.538332] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9276.899969] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9278.304220] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9279.916773] Lustre: Failing over lustre-MDT0000 [ 9280.077116] Lustre: server umount lustre-MDT0000 complete [ 9280.481941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9280.493552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9280.502544] Lustre: Skipped 1 previous similar message [ 9280.513255] LustreError: 347652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.912102] LustreError: 349194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.928697] LustreError: 349194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9295.138545] LustreError: 346597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9295.157747] LustreError: 346597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9297.026589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.103694] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9297.276824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.307352] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9300.229740] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9300.311350] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9302.512512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9302.543037] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9302.581556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9307.190725] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9308.304696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9311.428677] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9312.979135] Lustre: Failing over lustre-MDT0000 [ 9313.178498] Lustre: server umount lustre-MDT0000 complete [ 9315.643803] LustreError: 350378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.657849] LustreError: 350378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9317.858190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.868691] Lustre: Skipped 4 previous similar messages [ 9317.875518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.697210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.838324] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.067606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.109883] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9335.106807] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9336.099972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9336.303812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9336.308541] Lustre: Skipped 2 previous similar messages [ 9336.335425] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9336.370795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9343.785157] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9345.186992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9351.648792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.659632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9351.688387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.700788] Lustre: Skipped 4 previous similar messages [ 9356.595735] Lustre: server umount lustre-MDT0000 complete [ 9356.773644] LustreError: 346601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.798614] LustreError: 346601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 9359.792663] LustreError: 346582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068264 with bad export cookie 2987161779094536655 [ 9359.803020] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.803694] LustreError: 346582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9360.047473] Lustre: server umount lustre-MDT0001 complete [ 9373.481994] Lustre: server umount lustre-OST0000 complete [ 9385.053472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.943212] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9396.126271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.404420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9396.409029] Lustre: Skipped 1 previous similar message [ 9401.576783] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9408.255449] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9415.162578] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9424.437704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9430.391490] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9430.906299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9436.388817] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9446.373828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.379980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9446.381069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.384121] Lustre: Skipped 3 previous similar messages [ 9446.402507] LustreError: Skipped 1 previous similar message [ 9450.994764] Lustre: server umount lustre-OST0000 complete [ 9456.611970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9456.616847] Lustre: Skipped 2 previous similar messages [ 9466.850817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9466.857811] Lustre: Skipped 3 previous similar messages [ 9468.383185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9468.644659] Lustre: server umount lustre-MDT0000 complete [ 9471.969856] LustreError: 353663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.983238] LustreError: 353663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9472.087911] LustreError: 352917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068377 with bad export cookie 2987161779094537404 [ 9472.095326] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.096575] LustreError: 352917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9472.462083] Lustre: server umount lustre-MDT0001 complete [ 9483.224384] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 9485.903815] Key type lgssc unregistered [ 9486.164778] LNet: 356470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9486.177548] LNetError: 356470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9486.200153] LNet: Removed LNI 192.168.206.103@tcp [ 9487.051633] Key type .llcrypt unregistered [ 9487.057584] Key type ._llcrypt unregistered [ 9510.327529] Key type ._llcrypt registered [ 9510.331719] Key type .llcrypt registered [ 9510.473701] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9511.710829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9511.779204] alg: No test for adler32 (adler32-zlib) [ 9512.745741] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 9512.952290] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 9514.633560] Key type lgssc registered [ 9515.557680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9522.520562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9522.540367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9524.023816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.945927] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9533.410832] LustreError: 357967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.422460] LustreError: 357967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9534.510287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.781628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9538.856867] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9547.761451] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9555.007293] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9565.693297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.076435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.124883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9574.880011] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9583.632603] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9594.850915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.858352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9594.870347] Lustre: Skipped 2 previous similar messages [ 9595.631570] Lustre: server umount lustre-MDT0000 complete [ 9599.149312] LustreError: 357946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068504 with bad export cookie 14792111413196088742 [ 9599.150313] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9599.169214] LustreError: 357946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9599.922038] Lustre: server umount lustre-MDT0001 complete [ 9613.483958] Lustre: server umount lustre-OST0000 complete [ 9625.288255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9625.761336] LustreError: 361679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.860136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9628.904571] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9631.202031] LustreError: 361680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9635.301623] LustreError: 361679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.864698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.183781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9640.686355] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9646.860498] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9651.929486] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9658.134402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.417582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.529791] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9670.719283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9671.355607] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9683.429567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9683.434310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.441741] Lustre: Skipped 2 previous similar messages [ 9683.445163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.987641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.994122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9686.001371] Lustre: Skipped 1 previous similar message [ 9686.028902] Lustre: Skipped 2 previous similar messages [ 9689.647088] Lustre: server umount lustre-MDT0000 complete [ 9691.110994] LustreError: 361680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.126050] LustreError: 361680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9692.949101] LustreError: 361660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068597 with bad export cookie 14792111413196089855 [ 9692.964060] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9692.966495] LustreError: 361660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9693.352566] Lustre: server umount lustre-MDT0001 complete [ 9707.731197] Lustre: server umount lustre-OST0000 complete [ 9723.226796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.601346] LustreError: 365593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9723.666325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9728.209943] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9736.569552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.945766] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9750.370100] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9757.202843] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.744503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.036437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.043169] Lustre: Skipped 1 previous similar message [ 9768.143731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9770.429541] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9777.957314] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9783.776425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9783.780368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.785868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.851115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.968770] Lustre: server umount lustre-OST0000 complete [ 9797.089059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.094225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9797.119500] Lustre: Skipped 2 previous similar messages [ 9797.145751] Lustre: Skipped 3 previous similar messages [ 9799.899549] Lustre: server umount lustre-MDT0000 complete [ 9802.208269] LustreError: 366321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.240046] LustreError: 366321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9802.899981] LustreError: 365573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068707 with bad export cookie 14792111413196090877 [ 9802.903678] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9802.909671] LustreError: 365573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9803.135576] Lustre: server umount lustre-MDT0001 complete [ 9814.099885] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 9816.933653] Key type lgssc unregistered [ 9817.224692] LNet: 369128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9817.234295] LNetError: 369128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9817.247925] LNet: Removed LNI 192.168.206.103@tcp [ 9818.030248] Key type .llcrypt unregistered [ 9818.031749] Key type ._llcrypt unregistered [ 9841.769501] Key type ._llcrypt registered [ 9841.772709] Key type .llcrypt registered [ 9841.899478] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9843.203762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9843.273251] alg: No test for adler32 (adler32-zlib) [ 9844.353769] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 9844.666159] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 9846.353065] Key type lgssc registered [ 9847.362815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9855.537099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9855.572100] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.582584] LustreError: 370573:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9855.603050] LustreError: 370573:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9855.612349] LustreError: 370573:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9855.620682] LustreError: 370573:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9855.628794] LustreError: 370573:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9862.639240] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9862.643152] LustreError: 370970:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9862.657321] LustreError: 370970:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9862.662724] LustreError: 370970:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9862.668381] LustreError: 370970:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9862.691711] LustreError: 370970:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9882.147752] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [ 9884.966976] Key type lgssc unregistered [ 9885.246971] LNet: 371977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9885.262810] LNetError: 371977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9885.287078] LNet: Removed LNI 192.168.206.103@tcp [ 9886.062427] Key type .llcrypt unregistered [ 9886.068264] Key type ._llcrypt unregistered [ 9901.541485] Key type ._llcrypt registered [ 9901.543690] Key type .llcrypt registered [ 9901.711226] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [ 9917.985386] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [ 9919.213857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9919.231711] alg: No test for adler32 (adler32-zlib) [ 9920.367512] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 9920.632955] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [ 9922.343224] Key type lgssc registered [ 9923.455145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9948.969354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9948.990413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.216756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9950.247889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9950.343263] Lustre: lustre-MDT0000: new disk, initializing [ 9950.456390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.469527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9953.977834] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9962.952361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.066620] Lustre: 375132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9963.147640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9963.155744] Lustre: Skipped 1 previous similar message [ 9963.260764] Lustre: lustre-MDT0001: new disk, initializing [ 9963.387542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.437658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9963.452481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9967.903217] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9976.592257] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9982.284205] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9988.419951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.663147] Lustre: lustre-OST0000: new disk, initializing [ 9988.670145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9988.678254] Lustre: 377102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9988.780547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9993.939579] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [ 9996.855411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9996.864839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9996.981847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10001.996876] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10005.716477] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10005.956131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10010.099733] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10010.480265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10012.645078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10012.649308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10012.663728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10012.673460] Lustre: Skipped 1 previous similar message [10012.684177] Lustre: Skipped 1 previous similar message [10017.766095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10017.777209] Lustre: Skipped 1 previous similar message [10018.448064] Lustre: server umount lustre-OST0000 complete [10022.886336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.908855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.928172] Lustre: Skipped 1 previous similar message [10028.008954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.017944] Lustre: Skipped 1 previous similar message [10028.299546] Lustre: server umount lustre-MDT0000 complete [10031.895941] LustreError: 375125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068936 with bad export cookie 9746258067349135611 [10031.912151] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10032.209573] Lustre: server umount lustre-MDT0001 complete [10040.107909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10041.665436] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:15:45 (1786068945) [10047.461228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.936521] LustreError: 379560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.956038] LustreError: 379560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10047.999706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10052.147328] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10053.088946] LustreError: 379561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.184861] LustreError: 379560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.391114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.718046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10065.839605] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10073.179690] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10078.736450] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10086.481194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10086.968614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10093.914285] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10100.525859] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10106.760995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10106.976834] Lustre: lustre-OST0001: new disk, initializing [10106.979894] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10106.986496] Lustre: 382396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10107.066111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10113.342024] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10115.146640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10115.155237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10115.242406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10122.020911] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10125.800432] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10125.812243] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.827055] Lustre: Skipped 1 previous similar message [10125.833774] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10125.840024] Lustre: Skipped 1 previous similar message [10131.372220] Lustre: server umount lustre-OST0001 complete [10133.788390] LustreError: 381379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.172287] LustreError: 381381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.206368] LustreError: 381381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10149.350310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10149.356527] LustreError: Skipped 1 previous similar message [10149.363714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.373256] Lustre: Skipped 1 previous similar message [10149.380152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10149.385809] Lustre: Skipped 3 previous similar messages [10151.407627] LustreError: 381379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.428278] LustreError: 381379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10153.857264] Lustre: server umount lustre-OST0000 complete [10158.050571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10158.065808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.093670] Lustre: Skipped 1 previous similar message [10166.753436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.767560] Lustre: Skipped 6 previous similar messages [10172.383136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10172.524656] Lustre: server umount lustre-MDT0000 complete [10176.251027] LustreError: 379541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069081 with bad export cookie 9746258067349136493 [10176.261923] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.264308] LustreError: 379541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10182.625391] Lustre: server umount lustre-MDT0001 complete [10192.986489] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10196.275835] Key type lgssc unregistered [10196.596559] LNet: 384386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10196.614003] LNetError: 384386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10196.636606] LNet: Removed LNI 192.168.206.103@tcp [10197.492786] Key type .llcrypt unregistered [10197.496853] Key type ._llcrypt unregistered [10245.969490] Key type ._llcrypt registered [10245.972112] Key type .llcrypt registered [10246.069970] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10247.049384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10247.255122] alg: No test for adler32 (adler32-zlib) [10248.275687] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10248.527466] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10250.231150] Key type lgssc registered [10251.205030] Lustre: Echo OBD driver; http://www.lustre.org/ [10257.833530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10257.851558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.087650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10259.110234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10259.322450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10263.415027] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10270.288191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10270.360458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10270.386799] Lustre: 387198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10270.579190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10274.261665] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10279.626563] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10284.011720] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.537694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.545911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.546404] Lustre: Skipped 1 previous similar message [10285.556970] Lustre: Skipped 1 previous similar message [10290.656431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10290.659724] Lustre: Skipped 1 previous similar message [10291.734633] Lustre: server umount lustre-MDT0000 complete [10294.759210] LustreError: 387189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069199 with bad export cookie 16419893237252649608 [10294.765792] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10294.778281] LustreError: 387189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10295.092364] Lustre: server umount lustre-MDT0001 complete [10302.343123] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:20:06 (1786069206) [10307.855289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.259412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10312.331787] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10313.697494] LustreError: 389709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.729050] LustreError: 389709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10318.817292] LustreError: 389710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.318360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10320.667422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10325.291846] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10332.894328] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10338.281548] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10344.369377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.560790] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10344.762153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10351.207872] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10358.127673] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10364.854351] Lustre: Permanently deactivating lustre-OST0000 [10364.874893] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10372.649135] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10373.590778] LustreError: 392694:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10373.607626] LustreError: 392694:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10373.615716] LustreError: 392694:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10376.395968] LustreError: 392742:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10376.402775] LustreError: 392742:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10384.437097] Lustre: server umount lustre-OST0000 complete [10386.810853] LustreError: 392991:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10386.814403] LustreError: 392991:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10388.962508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10388.983989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10388.994728] Lustre: Skipped 1 previous similar message [10394.086871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.226410] Lustre: server umount lustre-MDT0000 complete [10399.205915] LustreError: 390720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.224714] LustreError: 390720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10402.264957] LustreError: 389691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069307 with bad export cookie 16419893237252650189 [10402.267444] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10402.271888] LustreError: 389691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10412.649431] Lustre: server umount lustre-MDT0001 complete [10422.227722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.340281] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10429.677476] LustreError: 394151:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10435.852173] Lustre: 394490:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.103@tcp [10438.186653] Lustre: server umount MGS complete [10449.132718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.575056] LustreError: 395214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10449.594712] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10449.600564] Lustre: Skipped 1 previous similar message [10449.669601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.578350] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10455.008579] LustreError: 395213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.735628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.013491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10467.431816] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10475.191506] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10481.250848] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10483.000915] Lustre: Permanently reactivating lustre-OST0000 [10483.022886] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10483.029482] Lustre: Skipped 2 previous similar messages [10512.352411] LustreError: 397042:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10519.338269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10519.680799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10519.689628] Lustre: Skipped 1 previous similar message [10525.856263] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10532.078626] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10537.445188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10537.459136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.481969] Lustre: Skipped 1 previous similar message [10537.489087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10537.504095] Lustre: Skipped 1 previous similar message [10538.468870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10543.073391] Lustre: server umount lustre-OST0000 complete [10548.710971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.737628] Lustre: Skipped 1 previous similar message [10548.748753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10548.756678] Lustre: Skipped 2 previous similar messages [10552.893733] Lustre: server umount lustre-MDT0000 complete [10553.827831] LustreError: 395214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.855521] LustreError: 395214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10556.808561] LustreError: 396788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069461 with bad export cookie 16419893237252651232 [10556.810900] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10556.818155] LustreError: 396788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10557.325463] Lustre: server umount lustre-MDT0001 complete [10569.253694] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10572.077564] Key type lgssc unregistered [10572.509562] LNet: 399678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10572.517369] LNetError: 399678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10572.539673] LNet: Removed LNI 192.168.206.103@tcp [10573.397848] Key type .llcrypt unregistered [10573.401782] Key type ._llcrypt unregistered [10591.927896] Key type ._llcrypt registered [10591.930246] Key type .llcrypt registered [10592.017328] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [10605.842378] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10606.774301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10606.901102] alg: No test for adler32 (adler32-zlib) [10608.008283] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10608.285666] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10610.023163] Key type lgssc registered [10611.221840] Lustre: Echo OBD driver; http://www.lustre.org/ [10640.951926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10640.963251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10642.159848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10642.194808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10642.266412] Lustre: lustre-MDT0000: new disk, initializing [10642.336420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10642.352601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10646.477587] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10655.936751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.062986] Lustre: 402833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10656.147329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10656.157738] Lustre: Skipped 1 previous similar message [10656.269649] Lustre: lustre-MDT0001: new disk, initializing [10656.386422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10656.420234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10656.436827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10660.469179] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10669.314986] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10674.157470] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10680.272544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10680.453282] Lustre: lustre-OST0000: new disk, initializing [10680.456605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10680.467934] Lustre: 404797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10680.565522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10687.367790] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10687.637263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10687.653426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10687.791970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10697.093972] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10701.198787] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10701.421082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10705.553422] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10705.797851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10708.454133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10708.469928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10708.487495] LustreError: Skipped 1 previous similar message [10708.490901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10708.530691] Lustre: Skipped 1 previous similar message [10713.552818] Lustre: server umount lustre-OST0000 complete [10717.664854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.676931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.703224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10717.711608] Lustre: Skipped 1 previous similar message [10718.689348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.708886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10718.717031] Lustre: Skipped 1 previous similar message [10722.993895] Lustre: server umount lustre-MDT0000 complete [10726.302371] LustreError: 403627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069631 with bad export cookie 8246999378859081573 [10726.311514] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10726.315509] LustreError: 403627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10726.643510] Lustre: server umount lustre-MDT0001 complete [10733.688320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10735.122284] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:27:18 (1786069638) [10741.359742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10741.742843] LustreError: 407257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10741.760474] LustreError: 407257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10741.797663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10745.671910] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10746.854389] LustreError: 407258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.969604] LustreError: 407257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10753.887168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.517384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10758.969471] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10766.578825] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10772.464326] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10777.825414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.227936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10783.384780] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10788.900980] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10790.426542] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10793.683412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10800.097512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10800.111653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10800.125312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10802.159506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10803.832996] Lustre: server umount lustre-OST0000 complete [10807.270434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.270717] LustreError: 401057:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000084603ff7 x1872829625732992/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10807.286381] Lustre: Skipped 1 previous similar message [10807.323905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10807.330814] Lustre: Skipped 2 previous similar messages [10813.764876] Lustre: server umount lustre-MDT0000 complete [10817.066812] LustreError: 407237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069722 with bad export cookie 8246999378859082455 [10817.070202] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10817.074607] LustreError: 407237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10817.380556] Lustre: server umount lustre-MDT0001 complete [10828.618190] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10831.288976] Key type lgssc unregistered [10831.572563] LNet: 410986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10831.576246] LNetError: 410986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10831.584118] LNet: Removed LNI 192.168.206.103@tcp [10832.271274] Key type .llcrypt unregistered [10832.273385] Key type ._llcrypt unregistered [10853.825673] Key type ._llcrypt registered [10853.828262] Key type .llcrypt registered [10853.913224] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10854.174192] Key type .llcrypt unregistered [10854.177110] Key type ._llcrypt unregistered [10870.560710] Key type ._llcrypt registered [10870.562292] Key type .llcrypt registered [10870.677695] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10871.864189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10871.881046] alg: No test for adler32 (adler32-zlib) [10872.992993] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10873.228932] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10874.903543] Key type lgssc registered [10875.768716] Lustre: Echo OBD driver; http://www.lustre.org/ [10882.781807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10882.803626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.141422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10888.350864] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10894.305833] LustreError: 413203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.330481] LustreError: 413203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10896.724817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10897.109307] LustreError: 413204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.158382] LustreError: 413204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10897.171713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10902.498468] LustreError: 413203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.525301] LustreError: 413203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10903.037495] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10907.623632] LustreError: 413200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.660484] LustreError: 413200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10909.491667] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10909.688589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10914.584646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.786346] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10914.795108] Lustre: Skipped 1 previous similar message [10914.897747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10918.569207] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [10927.642462] Lustre: server umount lustre-OST0000 complete [10933.734378] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10933.741333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.755510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.265150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.266681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.384610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.390793] Lustre: Skipped 1 previous similar message [10941.475086] Lustre: server umount lustre-MDT0000 complete [10944.783988] LustreError: 413183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069849 with bad export cookie 2295979497439598505 [10944.794235] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10954.390546] Lustre: server umount lustre-MDT0001 complete [10962.585981] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [10965.010845] Key type lgssc unregistered [10965.322562] LNet: 416353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10965.328182] LNetError: 416353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10965.345380] LNet: Removed LNI 192.168.206.103@tcp [10966.118093] Key type .llcrypt unregistered [10966.119933] Key type ._llcrypt unregistered [10986.170494] Key type ._llcrypt registered [10986.174432] Key type .llcrypt registered [10986.258348] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [10987.127731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10987.218037] alg: No test for adler32 (adler32-zlib) [10988.246271] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10988.447884] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [10990.135910] Key type lgssc registered [10991.300477] Lustre: Echo OBD driver; http://www.lustre.org/ [10997.696938] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10997.720375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.216051] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11018.335287] LustreError: 417722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.103@tcp: failed processing log, type 4: rc = -110 [11044.959665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11050.649348] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11057.418039] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11063.761398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.203169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11064.278197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11068.519663] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11069.410032] LustreError: 417747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.432268] LustreError: 417747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11074.534108] LustreError: 417748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.977874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.333117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11080.588536] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11088.402919] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11094.502840] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.181235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11102.195427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11102.213479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11104.184483] Lustre: server umount lustre-OST0000 complete [11108.841916] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11108.861346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.891814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11110.369967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11110.375868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.492172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.503896] Lustre: Skipped 2 previous similar messages [11117.589756] Lustre: server umount lustre-MDT0000 complete [11120.608868] LustreError: 418761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.627421] LustreError: 418761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11121.106267] LustreError: 417728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070026 with bad export cookie 7176241463975810492 [11121.124225] LustreError: 417728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11121.128810] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11121.379412] Lustre: server umount lustre-MDT0001 complete [11131.434608] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11134.146511] Key type lgssc unregistered [11134.420977] LNet: 421292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11134.432055] LNetError: 421292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11134.454113] LNet: Removed LNI 192.168.206.103@tcp [11135.373637] Key type .llcrypt unregistered [11135.375718] Key type ._llcrypt unregistered [11156.589766] Key type ._llcrypt registered [11156.591217] Key type .llcrypt registered [11156.764986] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11157.626148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11157.675104] alg: No test for adler32 (adler32-zlib) [11158.725921] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11159.012763] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11160.679280] Key type lgssc registered [11161.912514] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.807815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.824667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.344898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11176.371224] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11181.550248] LustreError: 422707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.565157] LustreError: 422707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11185.885533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.410061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11191.000841] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11198.708343] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11204.763185] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11211.903047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11212.353974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.448480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11218.767435] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11227.318519] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11232.738512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11232.745879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11232.774290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11235.134184] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.3@tcp (stopping) [11236.833843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.841523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11236.856224] Lustre: Skipped 1 previous similar message [11236.885608] Lustre: Skipped 1 previous similar message [11238.613660] Lustre: server umount lustre-MDT0000 complete [11240.220786] LustreError: 422703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.040393] LustreError: 422687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.3@tcp arrived at 1786070145 with bad export cookie 3316947736587176981 [11241.060788] LustreError: 422687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11241.957518] LustreError: 422706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.986983] LustreError: 422706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11242.833722] LustreError: 422686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070147 with bad export cookie 3316947736587176092 [11242.845462] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11247.073103] LustreError: 422707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.076898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11247.084158] LustreError: 422707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11247.110855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11247.115110] Lustre: Skipped 1 previous similar message [11252.193451] LustreError: 422703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.195418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11252.212277] LustreError: 422703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11257.311762] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11257.543252] Lustre: server umount lustre-OST0000 complete [11263.458776] LustreError: 422706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.488982] LustreError: 422706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11265.567249] LustreError: 423438:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001c3c054c x1872830202414336/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11265.885514] Lustre: server umount lustre-MDT0001 complete [11278.243556] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11281.296175] Key type lgssc unregistered [11281.642479] LNet: 426399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11281.649614] LNetError: 426399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11281.672916] LNet: Removed LNI 192.168.206.103@tcp [11282.562448] Key type .llcrypt unregistered [11282.567300] Key type ._llcrypt unregistered [11305.356524] Key type ._llcrypt registered [11305.358989] Key type .llcrypt registered [11305.488712] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11306.668690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11306.827465] alg: No test for adler32 (adler32-zlib) [11308.012203] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11308.266368] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11309.975290] Key type lgssc registered [11311.229911] Lustre: Echo OBD driver; http://www.lustre.org/ [11319.154315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11319.170378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11320.623869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11326.076933] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11331.042147] LustreError: 427818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.070572] LustreError: 427818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11335.036864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.393342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11339.315419] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11347.028819] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11352.528876] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11358.814793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11359.096867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11365.521864] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11366.279893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11373.863288] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11381.731556] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11381.741541] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11381.757847] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11381.768538] Lustre: Skipped 1 previous similar message [11385.155104] Lustre: server umount lustre-MDT0001 complete [11386.864503] LustreError: 428871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11386.902211] LustreError: 428871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11391.969651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11391.970860] LustreError: 427818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.976784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.979810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11391.990617] LustreError: 427818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11391.996880] Lustre: Skipped 2 previous similar messages [11397.090142] LustreError: 427817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.091121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11397.711751] Lustre: server umount lustre-OST0000 complete [11401.226742] Lustre: server umount lustre-MDT0000 complete [11414.094378] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11416.672118] Key type lgssc unregistered [11416.951552] LNet: 431502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11416.959825] LNetError: 431502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11416.979940] LNet: Removed LNI 192.168.206.103@tcp [11417.642857] Key type .llcrypt unregistered [11417.645387] Key type ._llcrypt unregistered [11440.639370] Key type ._llcrypt registered [11440.642289] Key type .llcrypt registered [11440.723323] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [11454.489946] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11455.214630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11455.333766] alg: No test for adler32 (adler32-zlib) [11456.344864] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11456.513205] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11458.175146] Key type lgssc registered [11459.144122] Lustre: Echo OBD driver; http://www.lustre.org/ [11490.991775] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11504.555226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11504.606303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11505.800153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11505.830303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11505.916762] Lustre: lustre-MDT0000: new disk, initializing [11506.004880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11506.021449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11510.356945] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11523.485156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11523.781890] Lustre: lustre-OST0000: new disk, initializing [11523.786569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11523.795960] Lustre: Skipped 1 previous similar message [11523.819606] Lustre: 436534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11523.964752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11529.723653] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11531.390759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11531.405158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11531.435130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11538.993787] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.464913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.693730] Lustre: 435515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11554.781204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11554.901172] Lustre: lustre-MDT0001: new disk, initializing [11555.071285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11555.141705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11555.153479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11559.915054] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11569.346590] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11569.593646] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11575.244477] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11575.593170] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11582.777340] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.3@tcp (stopping) [11582.949707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.966279] Lustre: Skipped 1 previous similar message [11586.839121] Lustre: server umount lustre-MDT0001 complete [11593.189955] LustreError: 435527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.210420] LustreError: 435527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11598.331370] LustreError: 435522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.340718] LustreError: 435522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11603.431878] LustreError: 435528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.462459] LustreError: 435528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11608.546988] LustreError: 435527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.548514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11608.585776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11608.608569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11608.613888] Lustre: Skipped 2 previous similar messages [11613.670469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11613.671528] LustreError: 435528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.787036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11619.295444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11619.733763] Lustre: server umount lustre-OST0000 complete [11630.053388] Lustre: server umount lustre-MDT0000 complete [11645.072763] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11648.198393] Key type lgssc unregistered [11648.577656] LNet: 440288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11648.594201] LNetError: 440288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11648.612316] LNet: Removed LNI 192.168.206.103@tcp [11649.461310] Key type .llcrypt unregistered [11649.463475] Key type ._llcrypt unregistered [11690.780891] Key type ._llcrypt registered [11690.782817] Key type .llcrypt registered [11690.959572] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [11691.281598] Key type .llcrypt unregistered [11691.285925] Key type ._llcrypt unregistered [11709.832420] Key type ._llcrypt registered [11709.838418] Key type .llcrypt registered [11709.941866] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [11711.154970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11711.205083] alg: No test for adler32 (adler32-zlib) [11712.399597] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11712.638853] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [11714.321921] Key type lgssc registered [11715.698701] Lustre: Echo OBD driver; http://www.lustre.org/ [11744.366205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11744.424701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.870356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11745.936312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11746.032615] Lustre: lustre-MDT0000: new disk, initializing [11746.150147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11746.183797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11751.027547] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11762.421350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.559202] Lustre: 443669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11762.599499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11762.603023] Lustre: Skipped 1 previous similar message [11762.701391] Lustre: lustre-MDT0001: new disk, initializing [11762.783313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11762.825467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11762.839807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11767.652106] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11776.087925] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11781.309044] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11788.223097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11788.653389] Lustre: lustre-OST0000: new disk, initializing [11788.657388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11788.673991] Lustre: 445636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11788.763720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11795.112268] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11798.125711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11798.137466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11798.304317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11804.407745] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11813.860225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.864996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.872474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.885585] Lustre: Skipped 1 previous similar message [11813.895205] Lustre: Skipped 2 previous similar messages [11817.358886] Lustre: server umount lustre-MDT0000 complete [11820.823887] LustreError: 444464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070725 with bad export cookie 11672439707092242827 [11820.826373] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11820.835873] LustreError: 444464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11821.226957] Lustre: server umount lustre-MDT0001 complete [11834.538496] Lustre: server umount lustre-OST0000 complete [11846.544067] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:45:49 (1786070749) [11853.319475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.829038] LustreError: 447665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.874876] LustreError: 447665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11853.952944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.747524] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11859.432144] LustreError: 447666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.525509] LustreError: 447665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.642917] LustreError: 447666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.974709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.236614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11874.231211] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11882.030300] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11887.972897] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11908.137630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11908.378046] Lustre: Found index 0 for lustre-OST0000, updating log [11908.600262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11914.702558] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [11922.273597] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11924.986356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11935.205707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11935.206868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11935.210900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.216396] LustreError: Skipped 1 previous similar message [11935.260404] Lustre: Skipped 2 previous similar messages [11936.295505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11936.302908] Lustre: Skipped 1 previous similar message [11939.161110] Lustre: server umount lustre-OST0000 complete [11956.193078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11956.199677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.218157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11956.224767] Lustre: Skipped 1 previous similar message [11961.326411] Lustre: server umount lustre-MDT0000 complete [11962.852202] LustreError: 448394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.878128] LustreError: 448394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11964.929989] LustreError: 449510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070869 with bad export cookie 11672439707092244220 [11964.934283] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11964.937886] LustreError: 449510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11965.414971] Lustre: server umount lustre-MDT0001 complete [11974.327504] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:47:58 (1786070878) [11975.903558] LNet: 1 local NIs in recovery (showing 1): 192.168.206.103@tcp [12591.730142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12592.277205] LustreError: 454079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12592.303981] LustreError: 454079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12592.330243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12595.573764] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12597.730148] LustreError: 454080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.848478] LustreError: 454079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.437417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.705032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.992559] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12616.040756] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12621.453401] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12641.075332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12641.399913] Lustre: Found index 0 for lustre-OST0000, updating log [12641.688148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12648.611811] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12653.792052] LNet: 1 local NIs in recovery (showing 1): 192.168.206.103@tcp [12658.249355] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12661.036173] Lustre: server umount lustre-OST0000 complete [12665.764198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.775019] Lustre: Skipped 1 previous similar message [12665.781986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12665.788831] Lustre: Skipped 3 previous similar messages [12666.851522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12666.860029] Lustre: Skipped 1 previous similar message [12667.873501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12667.898684] Lustre: Skipped 1 previous similar message [12669.920473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12669.935428] Lustre: Skipped 3 previous similar messages [12674.023241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12674.034811] Lustre: Skipped 7 previous similar messages [12679.135196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12679.305384] Lustre: server umount lustre-MDT0000 complete [12680.160949] LustreError: 454080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.193258] LustreError: 454080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12683.655705] LustreError: 454059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071588 with bad export cookie 11672439707092245305 [12683.658583] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12683.679068] LustreError: 454059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12684.222844] Lustre: server umount lustre-MDT0001 complete [12712.744876] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [12720.916233] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [12755.606294] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:00:58 (1786071658) [12772.124812] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [12774.582481] Key type lgssc unregistered [12774.856804] LNet: 460708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12774.863681] LNetError: 460708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12903.973596] LNet: Removed LNI 192.168.206.103@tcp [12904.806507] Key type .llcrypt unregistered [12904.812124] Key type ._llcrypt unregistered [12919.199529] Key type ._llcrypt registered [12919.203162] Key type .llcrypt registered [12919.325701] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [12920.301520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12920.317905] alg: No test for adler32 (adler32-zlib) [12921.391450] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [12921.685378] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [12923.416145] Key type lgssc registered [12924.694103] Lustre: Echo OBD driver; http://www.lustre.org/ [12929.021470] LNet: Added LNI 192.168.206.103@tcp100 [8/256/0/180] [12929.916621] LNet: Added LNI 192.168.206.103@tcp101 [8/256/0/180] [12930.653582] LNet: Added LNI 192.168.206.103@tcp102 [8/256/0/180] [12931.362578] LNet: Added LNI 192.168.206.103@tcp103 [8/256/0/180] [12932.159281] LNet: Added LNI 192.168.206.103@tcp104 [8/256/0/180] [12933.015302] LNet: Added LNI 192.168.206.103@tcp105 [8/256/0/180] [12933.808950] LNet: Added LNI 192.168.206.103@tcp106 [8/256/0/180] [12934.488803] LNet: Added LNI 192.168.206.103@tcp107 [8/256/0/180] [12935.279237] LNet: Added LNI 192.168.206.103@tcp108 [8/256/0/180] [12936.113976] LNet: Added LNI 192.168.206.103@tcp109 [8/256/0/180] [12936.816055] LNet: Added LNI 192.168.206.103@tcp110 [8/256/0/180] [12937.541445] LNet: Added LNI 192.168.206.103@tcp111 [8/256/0/180] [12938.427748] LNet: Added LNI 192.168.206.103@tcp112 [8/256/0/180] [12939.255551] LNet: Added LNI 192.168.206.103@tcp113 [8/256/0/180] [12940.169897] LNet: Added LNI 192.168.206.103@tcp114 [8/256/0/180] [12940.887915] LNet: Added LNI 192.168.206.103@tcp115 [8/256/0/180] [12941.565576] LNet: Added LNI 192.168.206.103@tcp116 [8/256/0/180] [12942.201129] LNet: Added LNI 192.168.206.103@tcp117 [8/256/0/180] [12942.803534] LNet: Added LNI 192.168.206.103@tcp118 [8/256/0/180] [12943.512438] LNet: Added LNI 192.168.206.103@tcp119 [8/256/0/180] [12944.192850] LNet: Added LNI 192.168.206.103@tcp120 [8/256/0/180] [12944.840654] LNet: Added LNI 192.168.206.103@tcp121 [8/256/0/180] [12945.369702] LNet: Added LNI 192.168.206.103@tcp122 [8/256/0/180] [12945.994893] LNet: Added LNI 192.168.206.103@tcp123 [8/256/0/180] [12946.684165] LNet: Added LNI 192.168.206.103@tcp124 [8/256/0/180] [12947.364939] LNet: Added LNI 192.168.206.103@tcp125 [8/256/0/180] [12948.031683] LNet: Added LNI 192.168.206.103@tcp126 [8/256/0/180] [12948.792767] LNet: Added LNI 192.168.206.103@tcp127 [8/256/0/180] [12949.430184] LNet: Added LNI 192.168.206.103@tcp128 [8/256/0/180] [12950.083755] LNet: Added LNI 192.168.206.103@tcp129 [8/256/0/180] [12950.719627] LNet: Added LNI 192.168.206.103@tcp130 [8/256/0/180] [12951.355296] LNet: Added LNI 192.168.206.103@tcp131 [8/256/0/180] [12952.119889] LNet: Added LNI 192.168.206.103@tcp132 [8/256/0/180] [12952.704186] LNet: Added LNI 192.168.206.103@tcp133 [8/256/0/180] [12953.284116] LNet: Added LNI 192.168.206.103@tcp134 [8/256/0/180] [12953.814118] LNet: Added LNI 192.168.206.103@tcp135 [8/256/0/180] [12959.380676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12959.432224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12960.566052] Lustre: 463719:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.103@tcp: can't add NID 192.168.206.103@tcp130: rc = -75 [12960.644112] LNetError: 463740:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12960.651518] LNetError: Protocol error connecting to 192.168.206.103@tcp106 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12960.660875] LNetError: Skipped 7 previous similar messages [12960.710831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12960.738737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12960.795558] Lustre: lustre-MDT0000: new disk, initializing [12960.891206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12960.926436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12961.638425] LNet: 11 local NIs in recovery (showing 11): 192.168.206.103@tcp122, 192.168.206.103@tcp123, 192.168.206.103@tcp125, 192.168.206.103@tcp126, 192.168.206.103@tcp127, 192.168.206.103@tcp129, 192.168.206.103@tcp128, 192.168.206.103@tcp124, 192.168.206.103@tcp130, 192.168.206.103@tcp131, 192.168.206.103@tcp132 [12961.640074] LNetError: 463743:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12961.683035] LNetError: 463743:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [12961.689576] LNetError: Protocol error connecting to 192.168.206.103@tcp115 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12961.695432] LNetError: Skipped 8 previous similar messages [12962.719966] LNetError: 463748:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12962.736886] LNetError: 463748:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [12962.743462] LNetError: Protocol error connecting to 192.168.206.103@tcp129 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12962.755497] LNetError: Skipped 4 previous similar messages [12965.045879] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12973.767872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.880442] Lustre: 463737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12973.933870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12973.941163] Lustre: Skipped 1 previous similar message [12974.004666] Lustre: lustre-MDT0001: new disk, initializing [12974.025951] LNetError: 461179:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12974.032597] LNetError: Protocol error connecting to 192.168.206.103@tcp133 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12974.036599] Lustre: 464536:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.103@tcp: can't add NID 192.168.206.103@tcp130: rc = -75 [12974.046954] Lustre: 464536:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12974.156110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12974.187463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12974.200221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12978.168778] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [12981.153717] LNetError: 461179:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12981.159544] LNetError: 461179:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [12981.164201] LNetError: Protocol error connecting to 192.168.206.103@tcp134 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12981.170014] LNetError: Skipped 5 previous similar messages [12981.793210] Lustre: 464965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.103@tcp: can't add NID 192.168.206.103@tcp130: rc = -75 [12981.799939] Lustre: 464965:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12985.164554] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12989.345633] LNetError: 463752:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [12989.368813] LNetError: Protocol error connecting to 192.168.206.103@tcp134 at host 192.168.206.103:988: is it running a compatible version of Lustre? [12989.624538] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12995.682942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12995.888065] Lustre: lustre-OST0000: new disk, initializing [12995.900916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12995.921742] Lustre: 465709:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12996.051671] Lustre: 465714:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.103@tcp: can't add NID 192.168.206.103@tcp130: rc = -75 [12996.058550] Lustre: 465714:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12996.107667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13002.778371] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13004.364522] Lustre: 466122:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.103@tcp: can't add NID 192.168.206.103@tcp130: rc = -75 [13004.387602] Lustre: 466122:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13004.506142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13004.534082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13004.688788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13005.728527] LNetError: 461177:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [13005.742477] LNetError: 461177:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [13005.762071] LNetError: Protocol error connecting to 192.168.206.103@tcp134 at host 192.168.206.103:988: is it running a compatible version of Lustre? [13005.780465] LNetError: Skipped 12 previous similar messages [13012.382431] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13020.129690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13020.131587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.142059] LustreError: Skipped 1 previous similar message [13020.142618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13020.150595] Lustre: Skipped 1 previous similar message [13020.178981] Lustre: Skipped 1 previous similar message [13021.087146] LNet: 1 local NIs in recovery (showing 1): 192.168.206.103@tcp134 [13024.734444] Lustre: server umount lustre-OST0000 complete [13030.368989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.371070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13030.374271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13030.381448] Lustre: Skipped 1 previous similar message [13033.997506] Lustre: server umount lustre-MDT0000 complete [13037.054409] LustreError: 463727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071942 with bad export cookie 16124980821739811164 [13037.063804] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13037.073467] LustreError: 463727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13037.410376] Lustre: server umount lustre-MDT0001 complete [13039.459449] LNetError: 463743:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.103 [13039.473800] LNetError: 463743:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 29 previous similar messages [13039.484896] LNetError: Protocol error connecting to 192.168.206.103@tcp120 at host 192.168.206.103:988: is it running a compatible version of Lustre? [13039.493220] LNetError: Skipped 29 previous similar messages [13047.161812] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [13049.271230] Key type lgssc unregistered [13049.529541] LNet: 467629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13049.536125] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13049.541873] LNet: Removed LNI 192.168.206.103@tcp [13049.545369] LNet: Removed LNI 192.168.206.103@tcp100 [13050.591751] LNet: Removed LNI 192.168.206.103@tcp101 [13050.602229] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13050.618077] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13050.622670] LNet: Removed LNI 192.168.206.103@tcp102 [13051.679125] LNet: Removed LNI 192.168.206.103@tcp103 [13051.683613] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13051.699346] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13051.708589] LNet: Removed LNI 192.168.206.103@tcp104 [13052.768551] LNet: Removed LNI 192.168.206.103@tcp105 [13053.792216] LNet: Removed LNI 192.168.206.103@tcp106 [13053.794831] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13053.800839] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13053.805143] LNet: Removed LNI 192.168.206.103@tcp107 [13054.816552] LNet: Removed LNI 192.168.206.103@tcp108 [13055.839164] LNet: Removed LNI 192.168.206.103@tcp109 [13056.863453] LNet: Removed LNI 192.168.206.103@tcp110 [13057.887313] LNet: Removed LNI 192.168.206.103@tcp111 [13057.889884] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13057.894967] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13057.907769] LNet: Removed LNI 192.168.206.103@tcp112 [13058.975304] LNet: Removed LNI 192.168.206.103@tcp113 [13059.999161] LNet: Removed LNI 192.168.206.103@tcp114 [13061.023497] LNet: Removed LNI 192.168.206.103@tcp115 [13062.047291] LNet: Removed LNI 192.168.206.103@tcp116 [13063.071572] LNet: Removed LNI 192.168.206.103@tcp117 [13064.097763] LNet: Removed LNI 192.168.206.103@tcp118 [13065.120045] LNet: Removed LNI 192.168.206.103@tcp119 [13066.143223] LNet: Removed LNI 192.168.206.103@tcp120 [13066.147201] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13066.152719] LNetError: 467629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13066.175055] LNet: Removed LNI 192.168.206.103@tcp121 [13067.231153] LNet: Removed LNI 192.168.206.103@tcp122 [13068.319284] LNet: Removed LNI 192.168.206.103@tcp123 [13069.344263] LNet: Removed LNI 192.168.206.103@tcp124 [13070.368339] LNet: Removed LNI 192.168.206.103@tcp125 [13071.391713] LNet: Removed LNI 192.168.206.103@tcp126 [13072.415114] LNet: Removed LNI 192.168.206.103@tcp127 [13073.439143] LNet: Removed LNI 192.168.206.103@tcp128 [13074.527217] LNet: Removed LNI 192.168.206.103@tcp129 [13075.551214] LNet: Removed LNI 192.168.206.103@tcp130 [13076.575162] LNet: Removed LNI 192.168.206.103@tcp131 [13077.599141] LNet: Removed LNI 192.168.206.103@tcp132 [13078.623391] LNet: Removed LNI 192.168.206.103@tcp133 [13078.627488] LNet: Removed LNI 192.168.206.103@tcp134 [13079.655907] LNet: Removed LNI 192.168.206.103@tcp135 [13080.348199] Key type .llcrypt unregistered [13080.351737] Key type ._llcrypt unregistered [13101.081425] Key type ._llcrypt registered [13101.083847] Key type .llcrypt registered [13101.207603] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [13116.799435] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [13118.114302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13118.163727] alg: No test for adler32 (adler32-zlib) [13119.354326] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [13119.623288] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [13121.287319] Key type lgssc registered [13122.856278] Lustre: Echo OBD driver; http://www.lustre.org/ [13154.444650] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:07:38 (1786072058) [13168.691798] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [13170.986741] Key type lgssc unregistered [13171.233936] LNet: 471722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13171.241667] LNetError: 471722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13171.255744] LNet: Removed LNI 192.168.206.103@tcp [13171.987160] Key type .llcrypt unregistered [13171.989514] Key type ._llcrypt unregistered [13185.341186] Key type ._llcrypt registered [13185.343518] Key type .llcrypt registered [13185.451768] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [13186.228912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13186.322035] alg: No test for adler32 (adler32-zlib) [13187.431582] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [13187.640556] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [13189.295409] Key type lgssc registered [13190.348691] Lustre: Echo OBD driver; http://www.lustre.org/ [13200.140225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13200.170808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13201.474137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13201.541681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13201.680770] Lustre: lustre-MDT0000: new disk, initializing [13201.749448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13201.767176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13206.247287] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13215.229037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13215.308577] Lustre: 472912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13215.364448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13215.369118] Lustre: Skipped 1 previous similar message [13215.444314] Lustre: lustre-MDT0001: new disk, initializing [13215.520433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13215.550920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13215.558334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13218.637114] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13227.108664] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13234.439829] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13244.138978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13244.575217] Lustre: lustre-OST0000: new disk, initializing [13244.579706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13244.585451] Lustre: 474881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13244.694192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13251.934234] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13252.638193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13252.643352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13252.806653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13260.559822] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13264.694738] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing lnet_if_list [13269.073141] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13269.107976] LNet: Added LNI 192.168.206.103@tcp10 [8/256/0/180] [13275.178262] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13275.203927] LNet: Added LNI 192.168.206.103@tcp11 [8/256/0/180] [13279.509147] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13279.546423] LNet: Added LNI 192.168.206.103@tcp12 [8/256/0/180] [13283.143168] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13283.162169] LNet: Added LNI 192.168.206.103@tcp13 [8/256/0/180] [13287.206067] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13287.228358] LNet: Added LNI 192.168.206.103@tcp14 [8/256/0/180] [13291.892317] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13291.933451] LNet: Added LNI 192.168.206.103@tcp15 [8/256/0/180] [13295.758208] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13295.791754] LNet: Added LNI 192.168.206.103@tcp16 [8/256/0/180] [13299.479496] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13299.501575] LNet: Added LNI 192.168.206.103@tcp17 [8/256/0/180] [13303.661386] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13303.707727] LNet: Added LNI 192.168.206.103@tcp18 [8/256/0/180] [13308.008151] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13308.037568] LNet: Added LNI 192.168.206.103@tcp19 [8/256/0/180] [13312.172956] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13312.194207] LNet: Added LNI 192.168.206.103@tcp20 [8/256/0/180] [13316.714895] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13316.743115] LNet: Added LNI 192.168.206.103@tcp21 [8/256/0/180] [13320.696414] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13320.718987] LNet: Added LNI 192.168.206.103@tcp22 [8/256/0/180] [13324.828519] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13324.855711] LNet: Added LNI 192.168.206.103@tcp23 [8/256/0/180] [13330.472625] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13330.513329] LNet: Added LNI 192.168.206.103@tcp24 [8/256/0/180] [13335.823067] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13335.858943] LNet: Added LNI 192.168.206.103@tcp25 [8/256/0/180] [13341.131369] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13341.170627] LNet: Added LNI 192.168.206.103@tcp26 [8/256/0/180] [13346.120152] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13346.164972] LNet: Added LNI 192.168.206.103@tcp27 [8/256/0/180] [13352.957703] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13353.001644] LNet: Added LNI 192.168.206.103@tcp28 [8/256/0/180] [13359.739091] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13359.801440] LNet: Added LNI 192.168.206.103@tcp29 [8/256/0/180] [13366.545095] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13366.581714] LNet: Added LNI 192.168.206.103@tcp30 [8/256/0/180] [13372.259203] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13372.279192] LNet: Added LNI 192.168.206.103@tcp31 [8/256/0/180] [13377.557526] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13377.595322] LNet: Added LNI 192.168.206.103@tcp32 [8/256/0/180] [13383.329198] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13383.353601] LNet: Added LNI 192.168.206.103@tcp33 [8/256/0/180] [13388.909748] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13388.946090] LNet: Added LNI 192.168.206.103@tcp34 [8/256/0/180] [13394.677347] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13394.698507] LNet: Added LNI 192.168.206.103@tcp35 [8/256/0/180] [13402.584099] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13402.629319] LNet: Added LNI 192.168.206.103@tcp36 [8/256/0/180] [13411.484341] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13411.560472] LNet: Added LNI 192.168.206.103@tcp37 [8/256/0/180] [13420.623710] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13420.650855] LNet: Added LNI 192.168.206.103@tcp38 [8/256/0/180] [13429.689671] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13429.741270] LNet: Added LNI 192.168.206.103@tcp39 [8/256/0/180] [13436.533309] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13436.556221] LNet: Added LNI 192.168.206.103@tcp40 [8/256/0/180] [13442.064560] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13442.095279] LNet: Added LNI 192.168.206.103@tcp41 [8/256/0/180] [13447.862303] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13447.882698] LNet: Added LNI 192.168.206.103@tcp42 [8/256/0/180] [13453.592412] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13453.644810] LNet: Added LNI 192.168.206.103@tcp43 [8/256/0/180] [13460.273764] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13460.300696] LNet: Added LNI 192.168.206.103@tcp44 [8/256/0/180] [13467.451952] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13467.490445] LNet: Added LNI 192.168.206.103@tcp45 [8/256/0/180] [13474.802291] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13474.840523] LNet: Added LNI 192.168.206.103@tcp46 [8/256/0/180] [13481.238702] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13481.276041] LNet: Added LNI 192.168.206.103@tcp47 [8/256/0/180] [13487.472101] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13487.498762] LNet: Added LNI 192.168.206.103@tcp48 [8/256/0/180] [13493.370737] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13493.397497] LNet: Added LNI 192.168.206.103@tcp49 [8/256/0/180] [13499.499725] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13499.531330] LNet: Added LNI 192.168.206.103@tcp50 [8/256/0/180] [13514.677582] Lustre: server umount lustre-MDT0000 complete [13517.660377] LustreError: 472906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072422 with bad export cookie 84610303354354466 [13517.664306] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13517.666551] LustreError: 472906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13517.898367] Lustre: server umount lustre-MDT0001 complete [13531.043238] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [13535.200484] Lustre: 472188:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786072424/real 1786072424] req@000000009db0bb4e x1872832330095744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786072440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13535.245380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13535.712110] Lustre: 472187:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786072424/real 1786072424] req@000000008fea8ee9 x1872832330095616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786072440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13535.746164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13537.207082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13538.271658] Lustre: 472186:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786072427/real 1786072427] req@000000004867b089 x1872832330096000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786072443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13543.391134] Lustre: 472186:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786072432/real 1786072433] req@00000000f7be8efb x1872832330096640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786072448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13543.421182] Lustre: 472186:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13544.424218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13545.447342] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13549.485416] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13549.540749] Lustre: 472188:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786072438/real 1786072438] req@0000000048e46bc5 x1872832330096768/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786072454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13549.591303] Lustre: 472188:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [13550.565609] LustreError: 499093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13550.609302] LustreError: 499093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13555.680724] LustreError: 499093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13555.712844] LustreError: 499093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13558.358910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13558.699554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13562.657537] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13563.890485] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13568.169710] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13573.312743] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13590.498603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13590.503398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13590.522491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13594.595911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13594.597171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13594.606634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13596.302908] Lustre: server umount lustre-OST0000 complete [13604.835535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13604.844968] Lustre: Skipped 1 previous similar message [13604.847402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13604.848290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13604.863059] Lustre: Skipped 2 previous similar messages [13606.083528] Lustre: server umount lustre-MDT0000 complete [13609.868644] LustreError: 473717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072514 with bad export cookie 84610303354357000 [13609.874671] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13609.878187] LustreError: 473717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13610.312305] Lustre: server umount lustre-MDT0001 complete [13618.912854] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [13621.460952] Key type lgssc unregistered [13621.967203] LNet: 501877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13621.976769] LNetError: 501877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13621.990754] LNet: Removed LNI 192.168.206.103@tcp [13622.007879] LNet: Removed LNI 192.168.206.103@tcp10 [13622.010952] LNet: Removed LNI 192.168.206.103@tcp11 [13622.023561] LNet: Removed LNI 192.168.206.103@tcp12 [13622.040731] LNet: Removed LNI 192.168.206.103@tcp13 [13622.050740] LNet: Removed LNI 192.168.206.103@tcp14 [13622.053127] LNet: Removed LNI 192.168.206.103@tcp15 [13622.055090] LNet: Removed LNI 192.168.206.103@tcp16 [13622.065629] LNet: Removed LNI 192.168.206.103@tcp17 [13622.071786] LNet: Removed LNI 192.168.206.103@tcp18 [13622.078979] LNet: Removed LNI 192.168.206.103@tcp19 [13622.090817] LNet: Removed LNI 192.168.206.103@tcp20 [13622.098797] LNet: Removed LNI 192.168.206.103@tcp21 [13622.107228] LNet: Removed LNI 192.168.206.103@tcp22 [13622.114378] LNet: Removed LNI 192.168.206.103@tcp23 [13622.128860] LNet: Removed LNI 192.168.206.103@tcp24 [13622.139557] LNet: Removed LNI 192.168.206.103@tcp25 [13622.146787] LNet: Removed LNI 192.168.206.103@tcp26 [13622.152756] LNet: Removed LNI 192.168.206.103@tcp27 [13622.156674] LNet: Removed LNI 192.168.206.103@tcp28 [13622.159887] LNet: Removed LNI 192.168.206.103@tcp29 [13622.163703] LNet: Removed LNI 192.168.206.103@tcp30 [13622.173391] LNet: Removed LNI 192.168.206.103@tcp31 [13622.175190] LNet: Removed LNI 192.168.206.103@tcp32 [13622.181353] LNet: Removed LNI 192.168.206.103@tcp33 [13622.183820] LNet: Removed LNI 192.168.206.103@tcp34 [13622.187726] LNet: Removed LNI 192.168.206.103@tcp35 [13622.191791] LNet: Removed LNI 192.168.206.103@tcp36 [13622.203902] LNet: Removed LNI 192.168.206.103@tcp37 [13622.212633] LNet: Removed LNI 192.168.206.103@tcp38 [13622.222835] LNet: Removed LNI 192.168.206.103@tcp39 [13622.227493] LNet: Removed LNI 192.168.206.103@tcp40 [13622.235443] LNet: Removed LNI 192.168.206.103@tcp41 [13622.242268] LNet: Removed LNI 192.168.206.103@tcp42 [13622.251785] LNet: Removed LNI 192.168.206.103@tcp43 [13622.256682] LNet: Removed LNI 192.168.206.103@tcp44 [13622.262853] LNet: Removed LNI 192.168.206.103@tcp45 [13622.267155] LNet: Removed LNI 192.168.206.103@tcp46 [13622.270766] LNet: Removed LNI 192.168.206.103@tcp47 [13622.274388] LNet: Removed LNI 192.168.206.103@tcp48 [13622.277957] LNet: Removed LNI 192.168.206.103@tcp49 [13622.288195] LNet: Removed LNI 192.168.206.103@tcp50 [13623.303684] Key type .llcrypt unregistered [13623.305398] Key type ._llcrypt unregistered [13640.286542] Key type ._llcrypt registered [13640.290570] Key type .llcrypt registered [13640.399138] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [13654.386895] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [13655.382613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13655.451934] alg: No test for adler32 (adler32-zlib) [13656.649486] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [13656.997288] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [13658.751670] Key type lgssc registered [13660.078824] Lustre: Echo OBD driver; http://www.lustre.org/ [13693.060402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13693.079355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13694.373671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13694.428426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13694.532764] Lustre: lustre-MDT0000: new disk, initializing [13694.638924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13694.667290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13698.992171] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13708.765547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13708.892652] Lustre: 505072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13708.950069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13708.955066] Lustre: Skipped 1 previous similar message [13709.061265] Lustre: lustre-MDT0001: new disk, initializing [13709.197427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13709.241556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13709.258458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13714.452868] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13723.677064] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13729.621496] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13737.464817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13737.612649] Lustre: *** cfs_fail_loc=90f, val=0*** [13737.667917] Lustre: lustre-OST0000: new disk, initializing [13737.671624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13737.676977] Lustre: 507086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13737.756946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13743.381449] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13744.731049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13744.738613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13744.844380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13751.600876] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13760.482984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13760.488522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13760.495580] LustreError: Skipped 1 previous similar message [13760.496322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13760.523181] Lustre: Skipped 1 previous similar message [13762.635756] Lustre: server umount lustre-OST0000 complete [13770.721104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13770.725215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13770.730530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13770.745641] Lustre: Skipped 1 previous similar message [13770.754713] Lustre: Skipped 1 previous similar message [13772.316625] Lustre: server umount lustre-MDT0000 complete [13775.475474] LustreError: 507096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072680 with bad export cookie 2406451857288728228 [13775.478803] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13775.481990] LustreError: 507096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13775.832996] Lustre: server umount lustre-MDT0001 complete [13783.369380] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [13785.268536] Key type lgssc unregistered [13785.483534] LNet: 508992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13785.489698] LNetError: 508992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13785.504812] LNet: Removed LNI 192.168.206.103@tcp [13786.056631] Key type .llcrypt unregistered [13786.058848] Key type ._llcrypt unregistered [13802.086138] Key type ._llcrypt registered [13802.090505] Key type .llcrypt registered [13802.275919] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [13804.086738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13804.147063] alg: No test for adler32 (adler32-zlib) [13805.531334] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [13805.807334] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [13807.519290] Key type lgssc registered [13809.227123] Lustre: Echo OBD driver; http://www.lustre.org/ [13818.329656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13818.344591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13819.653143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13823.982101] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13829.094457] LustreError: 510242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13829.123030] LustreError: 510242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13833.563632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13833.953640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13838.080033] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13845.888223] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13850.150443] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13855.240442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13855.492919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13860.289662] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13866.888236] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13876.319675] loop0: detected capacity change from 0 to 204800000 [13876.440419] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13876.451847] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13876.460425] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13881.623556] loop0: detected capacity change from 0 to 204800000 [13881.749274] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13881.763403] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13881.772979] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13886.226308] loop0: detected capacity change from 0 to 204800000 [13891.611945] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13891.748578] Lustre: fs15246-OST0000: new disk, initializing [13891.755705] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13891.761689] Lustre: 513835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13891.825180] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13898.133435] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13908.045595] loop1: detected capacity change from 0 to 204800000 [13912.678088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13912.817289] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13912.867610] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13912.942298] Lustre: fs15246-MDT0000: new disk, initializing [13913.058810] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13913.092771] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13913.226564] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13914.274278] Lustre: 514957:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000134a4bf4 x1872832978004864/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786072835 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13916.420330] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [13923.361406] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13923.375880] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13923.409144] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13941.742398] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.760826] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13943.662188] Lustre: server umount fs15246-MDT0000 complete [13946.855239] LustreError: 512024:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13951.831656] Lustre: server umount fs15246-OST0000 complete [13962.216932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13962.219416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13962.225291] Lustre: Skipped 1 previous similar message [13962.225960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13962.231607] Lustre: Skipped 1 previous similar message [13967.333343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13967.341397] Lustre: Skipped 1 previous similar message [13967.574678] Lustre: server umount lustre-OST0000 complete [13972.448218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13972.448848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13972.467754] Lustre: Skipped 1 previous similar message [13972.478822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13976.949469] Lustre: server umount lustre-MDT0000 complete [13977.579558] LustreError: 510261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13977.600256] LustreError: 510261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13980.219524] LustreError: 510222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072885 with bad export cookie 10858798575061097672 [13980.230946] LustreError: 510222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13980.237017] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13980.610183] Lustre: server umount lustre-MDT0001 complete [13989.392245] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [13991.872737] Key type lgssc unregistered [13992.200310] LNet: 518057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13992.207204] LNetError: 518057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13992.228824] LNet: Removed LNI 192.168.206.103@tcp [13993.169292] Key type .llcrypt unregistered [13993.174366] Key type ._llcrypt unregistered [14013.966472] Key type ._llcrypt registered [14013.970547] Key type .llcrypt registered [14014.080660] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [14014.951828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14015.035092] alg: No test for adler32 (adler32-zlib) [14016.051379] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [14016.222732] LNet: Added LNI 192.168.206.103@tcp [8/256/0/180] [14017.873640] Key type lgssc registered [14019.963888] Lustre: Echo OBD driver; http://www.lustre.org/ [14067.807312] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [14078.724824] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [14119.106771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14119.126561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14120.357215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14120.382990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14120.434137] Lustre: lustre-MDT0000: new disk, initializing [14120.514057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14120.554215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14124.809425] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [14135.068960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14135.159786] Lustre: 522534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14135.219572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14135.224538] Lustre: Skipped 1 previous similar message [14135.285730] Lustre: lustre-MDT0001: new disk, initializing [14135.402816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14135.479395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14135.491765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14141.591830] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [14150.867893] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14157.169463] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14164.438475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14164.888686] Lustre: lustre-OST0000: new disk, initializing [14164.891610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14164.896954] Lustre: 524506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14164.988838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14171.515653] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_default_debug -1 all [14172.288043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14172.320535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14172.569828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14180.588158] Lustre: DEBUG MARKER: oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14185.288615] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14185.519448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14190.009562] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14190.249429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14193.121984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14193.123983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14193.135221] LustreError: Skipped 1 previous similar message [14193.151691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14196.203219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14196.215050] Lustre: Skipped 2 previous similar messages [14198.039846] Lustre: server umount lustre-OST0000 complete [14201.313732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14201.323208] Lustre: Skipped 1 previous similar message [14201.333516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14201.349096] Lustre: Skipped 1 previous similar message [14206.452190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14206.474925] Lustre: Skipped 1 previous similar message [14207.659141] Lustre: server umount lustre-MDT0000 complete [14211.414984] LustreError: 522526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073116 with bad export cookie 11101894484415436073 [14211.428973] LustreError: MGC192.168.206.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14211.430869] LustreError: 522526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14211.749922] Lustre: server umount lustre-MDT0001 complete [14239.162950] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing set_hostid [14248.824210] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing load_modules_local [14277.839447] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14018 sec ========= 23:26:21 (1786073181) [14279.894904] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:26:22 (1786073182) === [14283.480744] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:26:26 (1786073186) === [14319.144875] Lustre: DEBUG MARKER: oleg603-server.virtnet: executing unload_modules_local [14322.779774] Key type lgssc unregistered [14323.144623] LNet: 531152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14323.150366] LNetError: 531152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14323.180244] LNet: Removed LNI 192.168.206.103@tcp [14325.030093] Key type .llcrypt unregistered [14325.032090] Key type ._llcrypt unregistered