[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515698420 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003105] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008636] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011145] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.023011] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028300] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033029] *** VALIDATE tmpfs *** [ 0.034468] *** VALIDATE proc *** [ 0.036182] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.039113] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042026] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.045722] debug: unmapping init [mem 0xffffffffb4859000-0xffffffffb4860fff] [ 0.048175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049644] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050022] ... version: 2 [ 0.051010] ... bit width: 48 [ 0.052011] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056010] ... event mask: 000000070000000f [ 0.057346] rcu: Hierarchical SRCU implementation. [ 0.059522] smp: Bringing up secondary CPUs ... [ 0.060626] x86: Booting SMP configuration: [ 0.061025] .... node #0, CPUs: #1 #2 #3 [ 0.067504] smp: Brought up 1 node, 4 CPUs [ 0.069022] smpboot: Max logical packages: 1 [ 0.070017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.120087] node 0 deferred pages initialised in 48ms [ 0.125008] devtmpfs: initialized [ 0.126289] x86/mm: Memory block size: 128MB [ 0.128832] gcov: version magic: 0x41383552 [ 0.131259] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133398] pinctrl core: initialized pinctrl subsystem [ 0.134498] [ 0.134954] ************************************************************* [ 0.135020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136025] ** ** [ 0.137028] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138015] ** ** [ 0.139010] ** This means that this kernel is built to expose internal ** [ 0.140013] ** IOMMU data structures, which may compromise security on ** [ 0.141011] ** your system. ** [ 0.142010] ** ** [ 0.143011] ** If you see this message and you are not debugging the ** [ 0.144012] ** kernel, report this immediately to your vendor! ** [ 0.145014] ** ** [ 0.146013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147016] ************************************************************* [ 0.148875] NET: Registered protocol family 16 [ 0.150433] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156123] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160055] cpuidle: using governor menu [ 0.162123] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164530] PCI: Using configuration type 1 for base access [ 0.166141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182051] cryptd: max_cpu_qlen set to 1000 [ 0.185323] ACPI: Added _OSI(Module Device) [ 0.187018] ACPI: Added _OSI(Processor Device) [ 0.189014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191035] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198031] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205118] ACPI: Interpreter enabled [ 0.206063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208014] ACPI: Using IOAPIC for interrupt routing [ 0.209116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.221000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238197] acpiphp: Slot [2] registered [ 0.240129] acpiphp: Slot [5] registered [ 0.241134] acpiphp: Slot [6] registered [ 0.243118] acpiphp: Slot [7] registered [ 0.244133] acpiphp: Slot [8] registered [ 0.246125] acpiphp: Slot [9] registered [ 0.248095] acpiphp: Slot [10] registered [ 0.249118] acpiphp: Slot [3] registered [ 0.251091] acpiphp: Slot [4] registered [ 0.252109] acpiphp: Slot [11] registered [ 0.253094] acpiphp: Slot [12] registered [ 0.254134] acpiphp: Slot [13] registered [ 0.256108] acpiphp: Slot [14] registered [ 0.258079] acpiphp: Slot [15] registered [ 0.259115] acpiphp: Slot [16] registered [ 0.261136] acpiphp: Slot [17] registered [ 0.262090] acpiphp: Slot [18] registered [ 0.264094] acpiphp: Slot [19] registered [ 0.265094] acpiphp: Slot [20] registered [ 0.267098] acpiphp: Slot [21] registered [ 0.268106] acpiphp: Slot [22] registered [ 0.270189] acpiphp: Slot [23] registered [ 0.272108] acpiphp: Slot [24] registered [ 0.273107] acpiphp: Slot [25] registered [ 0.275119] acpiphp: Slot [26] registered [ 0.276180] acpiphp: Slot [27] registered [ 0.278100] acpiphp: Slot [28] registered [ 0.280099] acpiphp: Slot [29] registered [ 0.281000] acpiphp: Slot [30] registered [ 0.281000] acpiphp: Slot [31] registered [ 0.283068] PCI host bridge to bus 0000:00 [ 0.284019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303328] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307462] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319960] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325115] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336632] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344928] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362984] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.377359] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.477386] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529034] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549605] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.589000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.601118] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.620017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.635017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645655] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.659013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.671014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.692000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.707081] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.709414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.714437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.716231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.721100] iommu: Default domain type: Passthrough [ 0.722000] SCSI subsystem initialized [ 0.722000] ACPI: bus type USB registered [ 0.723094] usbcore: registered new interface driver usbfs [ 0.725080] usbcore: registered new interface driver hub [ 0.726277] usbcore: registered new device driver usb [ 0.728218] pps_core: LinuxPPS API ver. 1 registered [ 0.730010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.734072] PTP clock support registered [ 0.736109] EDAC MC: Ver: 3.0.0 [ 0.738188] PCI: Using ACPI for IRQ routing [ 0.739580] NetLabel: Initializing [ 0.741013] NetLabel: domain hash size = 128 [ 0.742007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744079] NetLabel: unlabeled traffic allowed by default [ 0.748109] vgaarb: loaded [ 0.749331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751059] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759000] clocksource: Switched to clocksource kvm-clock [ 0.870631] VFS: Disk quotas dquot_6.6.0 [ 0.872350] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875031] *** VALIDATE ramfs *** [ 0.876231] *** VALIDATE hugetlbfs *** [ 0.878210] pnp: PnP ACPI init [ 0.880668] pnp: PnP ACPI: found 6 devices [ 0.915234] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917794] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919902] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922498] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.924955] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.927378] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.930440] NET: Registered protocol family 2 [ 0.933272] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939170] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943704] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.949622] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.953567] TCP: Hash tables configured (established 65536 bind 65536) [ 0.957383] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.961788] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965333] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968650] NET: Registered protocol family 1 [ 0.971301] RPC: Registered named UNIX socket transport module. [ 0.973460] RPC: Registered udp transport module. [ 0.975499] RPC: Registered tcp transport module. [ 0.977429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.979886] NET: Registered protocol family 44 [ 0.981739] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.984112] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.986439] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989524] PCI: CLS 0 bytes, default 64 [ 0.992123] Unpacking initramfs... [ 2.393697] debug: unmapping init [mem 0xffff9e62fcc54000-0xffff9e62fffbffff] [ 2.397986] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.400247] software IO TLB: mapped [mem 0x00000000a0c00000-0x00000000a4c00000] (64MB) [ 2.403224] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926398] Initialise system trusted keyrings [ 2.927736] Key type blacklist registered [ 2.929656] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.938396] zbud: loaded [ 2.941217] *** VALIDATE nfs *** [ 2.942468] *** VALIDATE nfs4 *** [ 2.943791] pstore: using deflate compression [ 2.947242] Platform Keyring initialized [ 3.066435] NET: Registered protocol family 38 [ 3.068229] Key type asymmetric registered [ 3.069963] Asymmetric key parser 'x509' registered [ 3.071766] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.074790] io scheduler mq-deadline registered [ 3.076724] io scheduler kyber registered [ 3.078768] io scheduler bfq registered [ 3.080622] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.083067] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.085283] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.087787] ACPI: Power Button [PWRF] [ 3.092983] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.100434] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.111137] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.117673] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.131062] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.159157] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.186716] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.190767] Non-volatile memory driver v1.3 [ 3.192216] Linux agpgart interface v0.103 [ 3.225816] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.228423] vda: detected capacity change from 0 to 74829824 [ 3.248401] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.251898] vdb: detected capacity change from 0 to 1073741824 [ 3.270685] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.273860] vdc: detected capacity change from 0 to 2621440000 [ 3.292897] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.297427] vdd: detected capacity change from 0 to 2621440000 [ 3.315438] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.318515] vde: detected capacity change from 0 to 4294967296 [ 3.354718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.358415] vdf: detected capacity change from 0 to 4294967296 [ 3.368490] libphy: Fixed MDIO Bus: probed [ 3.374257] usbcore: registered new interface driver usbserial_generic [ 3.377223] usbserial: USB Serial support registered for generic [ 3.380112] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.385609] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.388260] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.392699] mousedev: PS/2 mouse device common for all mice [ 3.395348] rtc_cmos 00:05: RTC can wake from S4 [ 3.397982] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.400850] rtc_cmos 00:05: registered as rtc0 [ 3.405763] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.409592] intel_pstate: CPU model not supported [ 3.412798] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.415809] hid: raw HID events driver (C) Jiri Kosina [ 3.420695] usbcore: registered new interface driver usbhid [ 3.423659] usbhid: USB HID core driver [ 3.425711] drop_monitor: Initializing network drop monitor service [ 3.426468] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.431532] Initializing XFRM netlink socket [ 3.437706] NET: Registered protocol family 10 [ 3.441507] Segment Routing with IPv6 [ 3.442896] NET: Registered protocol family 17 [ 3.445784] mpls_gso: MPLS GSO support [ 3.454130] RAS: Correctable Errors collector initialized. [ 3.456172] AVX version of gcm_enc/dec engaged. [ 3.457953] AES CTR mode by8 optimization enabled [ 3.541848] sched_clock: Marking stable (3541798394, 0)->(4555899849, -1014101455) [ 3.544698] registered taskstats version 1 [ 3.547281] Loading compiled-in X.509 certificates [ 3.549773] zswap: loaded using pool lzo/zbud [ 3.574064] Key type big_key registered [ 3.587503] Key type encrypted registered [ 3.589303] ima: No TPM chip found, activating TPM-bypass! [ 3.591467] ima: Allocated hash algorithm: sha1 [ 3.593308] ima: No architecture policies found [ 3.595100] evm: Initialising EVM extended attributes: [ 3.596849] evm: security.selinux [ 3.598120] evm: security.ima [ 3.599305] evm: security.capability [ 3.600777] evm: HMAC attrs: 0x1 [ 3.603164] rtc_cmos 00:05: setting system clock to 2026-08-25 02:13:27 UTC (1787624007) [ 3.609608] debug: unmapping init [mem 0xffffffffb5803000-0xffffffffb59fffff] [ 3.612558] debug: unmapping init [mem 0xffffffffb4582000-0xffffffffb4858fff] [ 3.621165] Write protecting the kernel read-only data: 28672k [ 3.624960] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.628326] debug: unmapping init [mem 0xffffffffb3514000-0xffffffffb35fffff] [ 3.664965] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.673849] systemd[1]: Detected virtualization kvm. [ 3.675827] systemd[1]: Detected architecture x86-64. [ 3.678271] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.703731] systemd[1]: No hostname configured. [ 3.705367] systemd[1]: Set hostname to . [ 3.707399] random: systemd: uninitialized urandom read (16 bytes read) [ 3.710857] systemd[1]: Initializing machine ID from random generator. [ 3.759482] random: ln: uninitialized urandom read (6 bytes read) [ 3.840454] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842406] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.846556] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.849764] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.464716] device-mapper: uevent: version 1.0.3 [ 4.467175] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.166436] virtio_net virtio0 ens2: renamed from eth0 [ 5.199300] random: fast init done [ 5.207133] scsi host0: ata_piix [ 5.278723] scsi host1: ata_piix [ 5.280968] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.283933] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.950846] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.106450] random: crng init done [ 10.108035] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.531649] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.700772] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.967882] SELinux: Disabled at runtime. [ 12.037666] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.047590] systemd[1]: Detected virtualization kvm. [ 12.049485] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.630793] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.636558] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.642329] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.647594] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.651395] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.661862] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.669059] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ 12.831714] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.195741] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.528663] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.553483] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.661419] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.674100] EDAC sbridge: Ver: 1.1.2 [ 15.492794] Key type dns_resolver registered [ 15.820312] NFS: Registering the id_resolver key type [ 15.822660] Key type id_resolver registered [ 15.824775] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 48.386447] libcfs: loading out-of-tree module taints kernel. [ 48.423158] Key type ._llcrypt registered [ 48.424885] Key type .llcrypt registered [ 48.493837] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 73.613372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 76.338890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.356038] alg: No test for adler32 (adler32-zlib) [ 78.536720] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 80.570038] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 82.744849] Key type lgssc registered [ 84.501517] hrtimer: interrupt took 4510617 ns [ 86.965794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.728549] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.831666] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 171.941612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.963981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.357708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.399444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.579579] Lustre: lustre-MDT0000: new disk, initializing [ 173.701491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.732372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.735301] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.220719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.385039] Lustre: 6540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.431639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.439874] Lustre: Skipped 1 previous similar message [ 194.582241] Lustre: lustre-MDT0001: new disk, initializing [ 194.709144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.764659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.784445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.635990] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.558853] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.615654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.942454] Lustre: lustre-OST0000: new disk, initializing [ 215.950929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.960902] Lustre: 8480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.036032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.742778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.769235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.939834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.689481] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.019375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.259128] Lustre: lustre-OST0001: new disk, initializing [ 242.264569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.277770] Lustre: 9555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.392896] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.954712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.968729] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.035745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 250.398558] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.341428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.837083] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.776945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.682883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 294.137561] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 296.956077] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 299.466677] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 301.004744] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 22:18:22 EDT 2026 [ 319.945395] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 321.961751] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.402090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.422387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.442740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.474970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.484941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.526505] Lustre: Skipped 1 previous similar message [ 344.767710] Lustre: server umount lustre-MDT0000 complete [ 349.021066] LustreError: 7488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624352 with bad export cookie 7521305787695132545 [ 349.026326] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.033018] LustreError: 7488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 351.714079] LustreError: 10335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.715547] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.745050] LustreError: 10335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 351.762234] Lustre: Skipped 2 previous similar messages [ 351.765377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.771058] Lustre: Skipped 1 previous similar message [ 354.721703] LustreError: 11106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.721839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 354.768145] LustreError: 11106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 355.592714] Lustre: server umount lustre-MDT0001 complete [ 360.144498] Lustre: server umount lustre-OST0000 complete [ 364.820785] Lustre: server umount lustre-OST0001 complete [ 372.967472] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 384.052292] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 423.146233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.421371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 423.488943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 423.645328] Lustre: lustre-MDT0000: new disk, initializing [ 423.752630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.771718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.132471] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 437.818639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.901725] Lustre: 15109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.003261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.011858] Lustre: Skipped 1 previous similar message [ 438.152930] Lustre: lustre-MDT0001: new disk, initializing [ 438.258825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.321123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.339634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.815655] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 453.421847] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 459.923489] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.708571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 470.311353] Lustre: lustre-OST0000: new disk, initializing [ 470.317861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 470.333568] Lustre: 17079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 470.513785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.983514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 471.995645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 472.240360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.841452] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 487.989865] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.619897] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 495.010356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 504.838286] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 505.149329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 513.001048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 513.008042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.051694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 513.067875] Lustre: Skipped 2 previous similar messages [ 513.131467] Lustre: Skipped 2 previous similar messages [ 516.114771] Lustre: server umount lustre-OST0000 complete [ 523.234675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.237199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 523.257108] Lustre: Skipped 1 previous similar message [ 529.791422] Lustre: server umount lustre-MDT0000 complete [ 533.476434] LustreError: 15120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.544828] LustreError: 15120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 536.964370] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624540 with bad export cookie 7521305787695134085 [ 536.984914] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 537.004070] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.749883] Lustre: server umount lustre-MDT0001 complete [ 542.888964] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:22:24 (1787624544) [ 545.369244] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 547.235335] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:22:29 (1787624549) [ 554.662469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.185512] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.219806] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 555.273128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.545585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 560.613163] LustreError: 19489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.853437] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.911284] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 574.793755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.247039] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 590.106501] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.502202] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.101277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 614.330485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 622.052250] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.401407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.409625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.421483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.447647] Lustre: Skipped 1 previous similar message [ 636.467406] Lustre: Skipped 4 previous similar messages [ 642.568682] Lustre: server umount lustre-OST0000 complete [ 652.352371] Lustre: server umount lustre-MDT0000 complete [ 656.307171] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624660 with bad export cookie 7521305787695134967 [ 656.313163] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.321717] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 656.975714] Lustre: server umount lustre-MDT0001 complete [ 669.577364] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 673.056435] Key type lgssc unregistered [ 673.453900] LNet: 23045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.461722] LNetError: 23045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.478805] LNet: Removed LNI 192.168.204.106@tcp [ 674.683789] Key type .llcrypt unregistered [ 674.685824] Key type ._llcrypt unregistered [ 698.559842] Key type ._llcrypt registered [ 698.566237] Key type .llcrypt registered [ 698.702612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 699.786824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.859377] alg: No test for adler32 (adler32-zlib) [ 700.878359] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 701.234220] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 702.912202] Key type lgssc registered [ 704.003824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.521786] loop: module loaded [ 714.597265] loop0: detected capacity change from 0 to 204800000 [ 714.723960] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.738587] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.975777] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.208339] loop0: detected capacity change from 0 to 204800000 [ 722.437460] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.452477] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.467151] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 728.427911] loop0: detected capacity change from 0 to 204800000 [ 734.847655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.871919] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.156274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 736.202771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 736.279590] Lustre: lustre-MDT0000: new disk, initializing [ 736.451994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.477895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 741.641819] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 750.847593] loop1: detected capacity change from 0 to 204800000 [ 757.521187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 757.898598] Lustre: lustre-OST07c6: new disk, initializing [ 757.903662] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 757.910160] Lustre: Skipped 1 previous similar message [ 757.914879] Lustre: 26242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 758.007336] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 765.186321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 765.198614] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 765.245333] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 765.678913] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 771.501410] Lustre: 25231:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 771.508488] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 771.515104] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 771.523102] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 771.529324] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 771.535996] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 787.908857] Lustre: 25230:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 787.924348] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.934403] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.940401] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 787.953768] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 787.967983] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.047356] Lustre: 25230:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 793.067683] Lustre: 25230:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.078640] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.086990] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.097658] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.114542] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.121863] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.139403] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.152977] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.169064] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.184769] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.197520] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.077469] Lustre: 25231:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 798.097641] Lustre: 25231:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 798.109514] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 798.121705] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.130420] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 798.142511] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.156902] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 798.174108] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.189806] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 798.203566] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.221109] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.230405] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.445921] Lustre: 25231:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 806.461244] Lustre: 25231:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 806.470791] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 806.484025] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.492518] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.505956] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.514221] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 806.531406] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.537988] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 806.549185] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.563348] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.572132] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 817.895485] Lustre: 26911:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 817.909863] Lustre: 26911:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 817.919683] Lustre: 26911:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 817.926181] Lustre: 26911:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.933833] Lustre: 26911:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.945446] Lustre: 26911:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.956538] Lustre: 26911:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 817.976607] Lustre: 26911:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.993431] Lustre: 26911:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.007243] Lustre: 26911:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 818.021512] Lustre: 26911:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.029133] Lustre: 26911:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 867.810897] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.854168] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 870.921158] Lustre: server umount lustre-OST07c6 complete [ 883.762397] Lustre: server umount lustre-MDT0000 complete [ 902.321853] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 905.256419] Key type lgssc unregistered [ 905.778892] LNet: 28805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.790833] LNetError: 28805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.804023] LNet: Removed LNI 192.168.204.106@tcp [ 906.490597] Key type .llcrypt unregistered [ 906.495711] Key type ._llcrypt unregistered [ 931.013749] Key type ._llcrypt registered [ 931.020827] Key type .llcrypt registered [ 931.166127] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 933.046548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.132295] alg: No test for adler32 (adler32-zlib) [ 934.476626] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 934.800967] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 936.553688] Key type lgssc registered [ 938.192454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 946.721598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 946.768666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.550200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.972627] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 958.946827] LustreError: 30082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.971896] LustreError: 30082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 964.066925] LustreError: 30081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.322587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.716230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.894808] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 977.522677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.460941] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 995.568155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.281503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.321952] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1014.965462] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1018.874395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1038.582848] Lustre: server umount lustre-OST0000 complete [ 1044.450275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.453755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.473952] Lustre: Skipped 1 previous similar message [ 1044.491895] Lustre: Skipped 1 previous similar message [ 1049.570224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.573889] Lustre: Skipped 1 previous similar message [ 1052.805610] Lustre: server umount lustre-MDT0000 complete [ 1054.691192] LustreError: 30076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.716580] LustreError: 30076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1055.712769] Lustre: 29388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787625043/real 1787625043] req@00000000989c4640 x1874460607024000/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787625059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1055.740733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.956526] LustreError: 30063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625060 with bad export cookie 16220561011320167582 [ 1056.961544] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.974550] LustreError: 30063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1057.391154] Lustre: server umount lustre-MDT0001 complete [ 1070.438494] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1073.812777] Key type lgssc unregistered [ 1074.097280] LNet: 33824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.119661] LNetError: 33824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.138697] LNet: Removed LNI 192.168.204.106@tcp [ 1075.346484] Key type .llcrypt unregistered [ 1075.348615] Key type ._llcrypt unregistered [ 1096.393596] Key type ._llcrypt registered [ 1096.398743] Key type .llcrypt registered [ 1096.512514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1097.169639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1097.208237] alg: No test for adler32 (adler32-zlib) [ 1098.200718] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1098.430880] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1100.072756] Key type lgssc registered [ 1101.231772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.378779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.391561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.875318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.669577] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1121.252134] LustreError: 35094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.303316] LustreError: 35094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1126.388810] LustreError: 35095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.952532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.635893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.224239] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1143.172425] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1148.467523] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1156.342101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.645776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.710369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1163.494612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1172.384381] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.324653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1188.338384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.354741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1190.371980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1190.372666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1190.386943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.592234] Lustre: server umount lustre-OST0000 complete [ 1200.626112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.639605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.654158] Lustre: Skipped 1 previous similar message [ 1200.669501] Lustre: Skipped 2 previous similar messages [ 1204.934686] Lustre: server umount lustre-MDT0000 complete [ 1205.736268] LustreError: 35095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.769283] LustreError: 35095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1208.960407] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625212 with bad export cookie 6088570354828763388 [ 1208.971410] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.974459] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1209.328335] Lustre: server umount lustre-MDT0001 complete [ 1221.227231] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1224.706515] Key type lgssc unregistered [ 1225.033273] LNet: 38652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.062973] LNetError: 38652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.090690] LNet: Removed LNI 192.168.204.106@tcp [ 1226.403981] Key type .llcrypt unregistered [ 1226.406307] Key type ._llcrypt unregistered [ 1253.625698] Key type ._llcrypt registered [ 1253.628433] Key type .llcrypt registered [ 1253.733490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1254.886076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.080753] alg: No test for adler32 (adler32-zlib) [ 1256.187957] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1256.512711] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1258.241496] Key type lgssc registered [ 1259.983707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1268.154785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1268.169628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.868214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.044561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1279.524287] LustreError: 40071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.559247] LustreError: 40071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1284.592368] LustreError: 40070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.938505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.552974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.162519] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1301.547634] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1306.750891] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1315.922890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.346659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.060802] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1333.143633] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1339.986522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1349.870277] Lustre: server umount lustre-MDT0000 complete [ 1350.121519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.127254] LustreError: 40066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.138909] Lustre: Skipped 2 previous similar messages [ 1350.160252] LustreError: 40066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1353.807996] LustreError: 41915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625357 with bad export cookie 14930083554340542928 [ 1353.817120] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.824987] LustreError: 41915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1354.329513] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 1354.363760] LustreError: 40066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.238598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.273440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1356.283776] Lustre: server umount lustre-MDT0001 complete [ 1368.793216] LustreError: 40052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.6@tcp arrived at 1787625372 with bad export cookie 14930083554340543593 [ 1388.520363] Lustre: server umount lustre-OST0000 complete [ 1407.531425] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1411.458935] Key type lgssc unregistered [ 1411.934528] LNet: 43961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.945215] LNetError: 43961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.978808] LNet: Removed LNI 192.168.204.106@tcp [ 1413.010479] Key type .llcrypt unregistered [ 1413.011768] Key type ._llcrypt unregistered [ 1437.829860] Key type ._llcrypt registered [ 1437.832420] Key type .llcrypt registered [ 1437.934722] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1439.169141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.286459] alg: No test for adler32 (adler32-zlib) [ 1440.244785] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1440.419473] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1442.088276] Key type lgssc registered [ 1443.801862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1453.776106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1453.805593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.455550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.828612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1465.833526] LustreError: 45382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.857725] LustreError: 45382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1470.950860] LustreError: 45383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.048539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.570042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.031473] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1485.559950] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1492.087455] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1498.915889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.252115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.272448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1505.056862] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1512.432571] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.362455] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1526.243729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.253311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.282113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.271543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.297328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.305789] Lustre: Skipped 1 previous similar message [ 1530.444103] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1535.560297] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1535.570915] Lustre: Skipped 2 previous similar messages [ 1536.996028] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1537.254750] Lustre: server umount lustre-OST0000 complete [ 1557.993196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.005650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.354763] Lustre: server umount lustre-MDT0000 complete [ 1564.132101] LustreError: 45382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.152217] LustreError: 45382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1564.887036] LustreError: 45364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625568 with bad export cookie 8529252724947241872 [ 1564.890915] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1564.901396] LustreError: 45364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1571.237141] Lustre: server umount lustre-MDT0001 complete [ 1584.528807] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1587.827869] Key type lgssc unregistered [ 1588.331253] LNet: 49092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.343329] LNetError: 49092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.379333] LNet: Removed LNI 192.168.204.106@tcp [ 1589.913656] Key type .llcrypt unregistered [ 1589.916979] Key type ._llcrypt unregistered [ 1612.193931] Key type ._llcrypt registered [ 1612.197786] Key type .llcrypt registered [ 1612.287490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1613.202468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1613.368286] alg: No test for adler32 (adler32-zlib) [ 1614.472905] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1614.643176] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1616.299537] Key type lgssc registered [ 1617.451421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1625.123956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.156205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.699219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.739430] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1636.836799] LustreError: 50510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.853840] LustreError: 50510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1639.848430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.467183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.528335] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1656.270501] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1662.024579] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1669.115075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.456783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.549558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1675.653690] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1683.321688] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.179133] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1694.260600] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1706.739523] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2422h41m47s [ 1708.312618] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1711.586188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.589789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.618107] Lustre: Skipped 2 previous similar messages [ 1711.625942] Lustre: Skipped 1 previous similar message [ 1712.681354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.690361] Lustre: Skipped 4 previous similar messages [ 1717.831140] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1717.849633] Lustre: Skipped 1 previous similar message [ 1723.993360] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1724.011597] Lustre: Skipped 3 previous similar messages [ 1725.408275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1725.739354] Lustre: server umount lustre-MDT0000 complete [ 1730.125220] LustreError: 51610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.584573] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1736.271515] LustreError: 50504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.296118] LustreError: 50504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1737.905472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.129239] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.407294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.474722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1739.440643] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1743.561583] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1746.817500] Lustre: DEBUG MARKER: Wait for df (36732) ... [ 1748.710694] Lustre: DEBUG MARKER: done [ 1754.081890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.092373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.114604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.128053] Lustre: Skipped 3 previous similar messages [ 1755.111621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.351521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.361872] Lustre: Skipped 5 previous similar messages [ 1767.392147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.566513] Lustre: server umount lustre-OST0000 complete [ 1775.588370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.594118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.601543] Lustre: Skipped 1 previous similar message [ 1777.505795] Lustre: server umount lustre-MDT0000 complete [ 1781.530589] LustreError: 50489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625785 with bad export cookie 9605047469320555145 [ 1781.532529] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.539438] LustreError: 50489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1781.994681] Lustre: server umount lustre-MDT0001 complete [ 1793.085523] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1796.369873] Key type lgssc unregistered [ 1796.728241] LNet: 55441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.737581] LNetError: 55441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.758321] LNet: Removed LNI 192.168.204.106@tcp [ 1797.675322] Key type .llcrypt unregistered [ 1797.677186] Key type ._llcrypt unregistered [ 1845.537180] Key type ._llcrypt registered [ 1845.539424] Key type .llcrypt registered [ 1845.676203] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1846.901705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.922809] alg: No test for adler32 (adler32-zlib) [ 1848.082916] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1848.281603] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1849.993238] Key type lgssc registered [ 1850.745125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1858.920683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1858.956276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.297280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1860.321409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1860.472839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.895831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1874.308563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.379595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1874.407893] Lustre: 57648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1874.787810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1880.345690] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1889.512916] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1894.933723] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1901.797256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.115366] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1902.302694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.166790] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1912.395913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1914.878152] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1920.865865] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2422h45m22s [ 1922.536582] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1930.040750] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1932.548961] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1932.584493] LustreError: 57655:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1952.909350] Lustre: lustre-MDT0000: Client 51415045-38f7-4718-8f42-3cc204fcf675 (at 192.168.204.6@tcp) reconnecting [ 1972.616595] LustreError: 57655:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1975.057110] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1977.834545] Lustre: DEBUG MARKER: done [ 1983.570680] Lustre: server umount lustre-OST0000 complete [ 1987.553719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1987.571731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.587455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.652497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.660100] Lustre: Skipped 2 previous similar messages [ 1994.729087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.736363] Lustre: Skipped 1 previous similar message [ 1997.952476] Lustre: server umount lustre-MDT0000 complete [ 1999.841631] Lustre: 56971:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787625987/real 1787625987] req@000000002b84b61e x1874461565440896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787626003 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.891234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.910036] Lustre: Skipped 1 previous similar message [ 2002.384244] LustreError: 58391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626006 with bad export cookie 8355255262683344209 [ 2002.387900] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.416094] LustreError: 58391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2002.928527] Lustre: server umount lustre-MDT0001 complete [ 2013.749419] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2016.426374] Key type lgssc unregistered [ 2016.679611] LNet: 61983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.683759] LNetError: 61983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.695829] LNet: Removed LNI 192.168.204.106@tcp [ 2017.290467] Key type .llcrypt unregistered [ 2017.292776] Key type ._llcrypt unregistered [ 2066.439523] Key type ._llcrypt registered [ 2066.442927] Key type .llcrypt registered [ 2066.534709] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2068.291226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.361038] alg: No test for adler32 (adler32-zlib) [ 2069.400641] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2069.839928] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2071.544352] Key type lgssc registered [ 2072.834873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2076.284886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.296106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.512707] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2077.542282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2077.711993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.659595] Lustre: Failing over lustre-MDT0000 [ 2078.905737] Lustre: server umount lustre-MDT0000 complete [ 2085.681072] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:48:07 (1787626087) [ 2092.811779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.216472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.028508] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2107.289775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.412386] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2107.440954] Lustre: 64782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.671981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.057889] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2119.995878] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2125.773878] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2132.532970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2132.753473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2133.020248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.872584] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2142.382854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2147.423607] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2152.349867] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2159.074226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.080100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.087162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.176754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.189612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.224237] Lustre: Skipped 1 previous similar message [ 2165.171757] Lustre: server umount lustre-MDT0000 complete [ 2169.879265] LustreError: 66647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626173 with bad export cookie 11966107463148127879 [ 2169.884144] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.890447] LustreError: 66647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2170.400607] Lustre: server umount lustre-MDT0001 complete [ 2174.243442] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2178.635931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2188.220176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.792475] Lustre: 63705:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626177/real 1787626177] req@00000000dde5cbe7 x1874461797163008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2189.841997] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.880287] Lustre: 63705:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626178/real 1787626178] req@00000000211b5daa x1874461797163136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.912225] Lustre: 63708:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626182/real 1787626182] req@000000008d357774 x1874461797163264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.260474] LustreError: 66630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.279590] LustreError: 66630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2195.346510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.472834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2198.369517] LustreError: 66631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.765306] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2199.411294] LustreError: 68552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.433769] LustreError: 68552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2200.419467] Lustre: 63706:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626188/real 1787626188] req@0000000029cc328f x1874461797163648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.436569] Lustre: 63706:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2204.643680] LustreError: 68554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.653492] LustreError: 68554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2205.664969] Lustre: 63706:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626193/real 1787626193] req@00000000a7da5e93 x1874461797163776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.413824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.934194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.094147] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2212.371955] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2218.552500] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2219.245643] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2226.861396] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.922800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2237.940470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.960719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.964694] Lustre: Skipped 2 previous similar messages [ 2240.498698] Lustre: server umount lustre-MDT0000 complete [ 2244.068076] LustreError: 68583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.092931] LustreError: 68583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2244.682681] LustreError: 64775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626248 with bad export cookie 11966107463148130154 [ 2244.684256] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.689026] LustreError: 64775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2244.939402] Lustre: server umount lustre-MDT0001 complete [ 2251.843474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2265.568164] Lustre: 63707:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787626253/real 1787626253] req@00000000542c1656 x1874461797192320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787626269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.617064] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.635291] Lustre: Skipped 2 previous similar messages [ 2267.180569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.472157] LustreError: 71223:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2269.482399] LustreError: 71223:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000031f4bc21 x1874461797192448/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787626273 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2269.515599] LustreError: 71223:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2269.736163] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa6102756ad1854cd [ 2269.760888] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2270.216135] LustreError: 66631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.376696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2270.502732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2275.982838] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2285.254370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.542346] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2285.557171] Lustre: Skipped 4 previous similar messages [ 2289.939100] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2291.173748] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2298.275860] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.571378] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2311.650379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.663534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.682582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.634709] Lustre: server umount lustre-MDT0000 complete [ 2314.727356] LustreError: 72279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.758341] LustreError: 72279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2315.852413] LustreError: 64776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626319 with bad export cookie 11966107463148131533 [ 2315.853496] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.859648] LustreError: 64776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2316.203593] Lustre: server umount lustre-MDT0001 complete [ 2323.005301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2326.880933] Lustre: DEBUG MARKER: files compared the same [ 2340.042594] Lustre: server umount lustre-OST0000 complete [ 2357.695904] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2361.257841] Key type lgssc unregistered [ 2361.647503] LNet: 74516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.658316] LNetError: 74516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.706724] LNet: Removed LNI 192.168.204.106@tcp [ 2363.088743] Key type .llcrypt unregistered [ 2363.090820] Key type ._llcrypt unregistered [ 2388.036140] Key type ._llcrypt registered [ 2388.037881] Key type .llcrypt registered [ 2388.153660] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2388.965931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2389.015486] alg: No test for adler32 (adler32-zlib) [ 2390.221872] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2390.478724] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2392.144773] Key type lgssc registered [ 2393.737734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.769755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.803794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.405132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.513661] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2413.538296] LustreError: 75933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.545900] LustreError: 75933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2417.134770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.604227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.276910] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2431.697287] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.478939] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2448.797153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.243429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.959965] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2465.373904] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2473.830923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2484.195070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.213681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.219326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.320513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.327509] Lustre: Skipped 1 previous similar message [ 2489.767532] Lustre: server umount lustre-OST0000 complete [ 2494.434504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.457541] Lustre: Skipped 1 previous similar message [ 2494.477098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.485659] Lustre: Skipped 2 previous similar messages [ 2499.552690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.562227] Lustre: Skipped 1 previous similar message [ 2499.777849] Lustre: server umount lustre-MDT0000 complete [ 2504.565822] LustreError: 75913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626508 with bad export cookie 6286574936017451876 [ 2504.568777] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.584742] LustreError: 75913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2505.360508] Lustre: server umount lustre-MDT0001 complete [ 2520.637576] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2523.535518] Key type lgssc unregistered [ 2523.887535] LNet: 79506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.893677] LNetError: 79506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.914574] LNet: Removed LNI 192.168.204.106@tcp [ 2524.611639] Key type .llcrypt unregistered [ 2524.613389] Key type ._llcrypt unregistered [ 2548.733215] Key type ._llcrypt registered [ 2548.735471] Key type .llcrypt registered [ 2548.894168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2550.187866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.363724] alg: No test for adler32 (adler32-zlib) [ 2551.505757] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2551.843528] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2553.553567] Key type lgssc registered [ 2554.876354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.950463] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.972616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.552601] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2584.674761] LustreError: 80899:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 2611.488186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.126221] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2625.470993] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2634.906909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.068879] Lustre: *** cfs_fail_loc=706, val=0*** [ 2635.525949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.639115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2640.870774] LustreError: 80925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.914628] LustreError: 80925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2642.518677] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2645.985930] LustreError: 80924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.031601] LustreError: 81982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.062573] LustreError: 81982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2652.132577] LustreError: 81986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.148853] LustreError: 81986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2653.231835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.514311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2658.760498] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2667.891439] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2674.847996] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2679.272628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.306739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.312646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.362284] Lustre: Skipped 1 previous similar message [ 2684.389369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.392872] Lustre: Skipped 1 previous similar message [ 2689.506670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.064667] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2692.206569] Lustre: server umount lustre-OST0000 complete [ 2699.745396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2699.746963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.753321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.753329] Lustre: Skipped 1 previous similar message [ 2699.786381] Lustre: Skipped 2 previous similar messages [ 2702.740777] Lustre: server umount lustre-MDT0000 complete [ 2704.867188] LustreError: 81990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.889540] LustreError: 81990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2706.315188] LustreError: 80906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626710 with bad export cookie 15996784595231886731 [ 2706.316500] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.329779] LustreError: 80906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2706.546920] Lustre: server umount lustre-MDT0001 complete [ 2717.277351] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2719.811540] Key type lgssc unregistered [ 2720.139988] LNet: 84521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.146699] LNetError: 84521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2720.159340] LNet: Removed LNI 192.168.204.106@tcp [ 2721.056078] Key type .llcrypt unregistered [ 2721.059889] Key type ._llcrypt unregistered [ 2746.746266] Key type ._llcrypt registered [ 2746.749521] Key type .llcrypt registered [ 2746.902867] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2748.067283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.096782] alg: No test for adler32 (adler32-zlib) [ 2749.141393] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2749.413827] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2751.201109] Key type lgssc registered [ 2752.499808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.394677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2762.413424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.624350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2777.730481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.332705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.225129] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2783.714731] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.735574] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2788.835535] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.868538] LustreError: 86624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2792.569314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2792.889634] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.922330] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2792.945282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.060567] LustreError: 87375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.085022] LustreError: 87375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2799.774627] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2803.172683] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.184678] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2809.422985] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.440768] Lustre: MGS: Client 5a141852-f1e1-4b4a-90ae-fc7ae055b8cd (at 0@lo) reconnecting [ 2809.472539] LustreError: 86639:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2809.718869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.780634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2814.711435] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2830.312489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2830.314335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.333241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2830.343349] Lustre: Skipped 1 previous similar message [ 2835.436823] LustreError: 87352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.454045] LustreError: 87352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2835.674146] Lustre: server umount lustre-OST0000 complete [ 2840.552665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2840.556229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.567727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.593960] Lustre: Skipped 1 previous similar message [ 2840.626833] Lustre: Skipped 2 previous similar messages [ 2845.666660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.678658] Lustre: Skipped 1 previous similar message [ 2846.001689] Lustre: server umount lustre-MDT0000 complete [ 2851.934651] Lustre: server umount lustre-MDT0001 complete [ 2857.323427] Lustre: server umount MGS complete [ 2874.522996] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:01:15 (1787626875) [ 2911.538978] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2924.901538] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2971.612042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.484353] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2985.249466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.467498] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2985.503316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2985.558494] Lustre: lustre-MDT0001: new disk, initializing [ 2985.624338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2985.664838] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2985.677069] Lustre: 93825:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2985.687745] LustreError: 93825:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2989.759943] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3001.032852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.409500] Lustre: lustre-OST0000: new disk, initializing [ 3001.415568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3001.426889] Lustre: 94593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3001.519419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.442318] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3009.060680] Lustre: 94907:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ad5085a5 x1874462510195200/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787627028 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3019.565775] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3028.115496] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.127289] Lustre: MGS: Client 288c47a6-6aec-488f-adfb-c50faf6880cd (at 0@lo) reconnecting [ 3028.166953] Lustre: 93095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3028.190410] LustreError: 93811:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3028.203336] LustreError: 93811:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3028.220403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3028.368948] Lustre: lustre-MDT0000: new disk, initializing [ 3028.536445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.595865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3028.619863] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3029.670445] Lustre: 95662:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002bd4e7d2 x1874462510202112/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787627049 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3033.419312] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3034.784826] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3036.679761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3036.689272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3038.777304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3038.796567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3038.836844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3064.801539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3064.811414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.831959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.362185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.716531] Lustre: server umount lustre-OST0000 complete [ 3077.602487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.608796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3077.612561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.630189] Lustre: Skipped 1 previous similar message [ 3077.660657] Lustre: Skipped 3 previous similar messages [ 3080.627883] Lustre: server umount lustre-MDT0000 complete [ 3082.721330] LustreError: 96087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.739754] LustreError: 96087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3090.999839] Lustre: server umount lustre-MDT0001 complete [ 3095.484833] Lustre: server umount MGS complete [ 3106.745796] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:05:08 (1787627108) [ 3126.969634] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3130.238669] Key type lgssc unregistered [ 3130.708214] LNet: 98163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.713945] LNetError: 98163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.730737] LNet: Removed LNI 192.168.204.106@tcp [ 3131.662513] Key type .llcrypt unregistered [ 3131.668662] Key type ._llcrypt unregistered [ 3151.672400] Key type ._llcrypt registered [ 3151.675768] Key type .llcrypt registered [ 3151.799898] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3152.990183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3153.035258] alg: No test for adler32 (adler32-zlib) [ 3154.243446] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3154.643747] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3156.360714] Key type lgssc registered [ 3157.474669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3166.321898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3166.351186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.715084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3172.215669] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3177.971523] LustreError: 99404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.010470] LustreError: 99404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3183.075158] LustreError: 99403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.763257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.305966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3190.261338] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3200.158662] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3206.655823] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3213.216550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.519813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.806149] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3229.161378] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.151403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3248.613975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.616040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.624750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3248.630856] Lustre: Skipped 1 previous similar message [ 3253.734543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3253.739145] Lustre: Skipped 1 previous similar message [ 3254.789473] Lustre: server umount lustre-OST0000 complete [ 3261.414757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3261.432061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.471573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.491669] Lustre: Skipped 1 previous similar message [ 3263.972629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.974082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.095675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.109238] Lustre: Skipped 2 previous similar messages [ 3274.209139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3274.579744] Lustre: server umount lustre-MDT0000 complete [ 3278.317988] LustreError: 100490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.350749] LustreError: 100490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3279.368117] LustreError: 99383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627283 with bad export cookie 12779672378186988829 [ 3279.384618] LustreError: 99383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3279.392762] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.849550] Lustre: server umount lustre-MDT0001 complete [ 3292.931114] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3296.546339] Key type lgssc unregistered [ 3296.856888] LNet: 102963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.864382] LNetError: 102963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.885868] LNet: Removed LNI 192.168.204.106@tcp [ 3297.986314] Key type .llcrypt unregistered [ 3297.990234] Key type ._llcrypt unregistered [ 3318.608622] Key type ._llcrypt registered [ 3318.611755] Key type .llcrypt registered [ 3318.735100] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3320.417914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3320.454143] alg: No test for adler32 (adler32-zlib) [ 3321.670539] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3321.934461] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3323.656188] Key type lgssc registered [ 3324.931854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3332.623191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3332.636688] LustreError: 104043:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3332.653983] LustreError: 104046:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3332.661086] LustreError: 104043:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3332.668422] LustreError: 104043:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3332.680694] LustreError: 104043:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3332.692734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.194078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.104922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.424316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3348.396682] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3355.067479] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lsmod [ 3358.438681] LustreError: 105252:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3358.465305] LustreError: 105253:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3358.470789] LustreError: 105252:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3358.478088] LustreError: 105252:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3358.491305] LustreError: 105252:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3358.520132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3358.961423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.999245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3363.812230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3363.829727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.859947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.345916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.348721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.359519] Lustre: Skipped 1 previous similar message [ 3365.383659] Lustre: Skipped 1 previous similar message [ 3368.146987] Lustre: server umount lustre-MDT0000 complete [ 3368.484752] LustreError: 105441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.499755] LustreError: 105441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3372.093897] LustreError: 104059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627375 with bad export cookie 15510916928569820098 [ 3372.098290] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.104449] LustreError: 104059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.600284] LustreError: 104515:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000663af88c x1874463109960832/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3372.851322] Lustre: server umount lustre-MDT0001 complete [ 3387.026414] Lustre: server umount lustre-OST0000 complete [ 3395.547147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.997985] LustreError: 106321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.122646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.191331] LustreError: 106321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.224426] LustreError: 106321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3401.502785] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3405.286108] LustreError: 106322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.402906] LustreError: 106321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.078439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.655303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.160540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3425.345772] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3431.903513] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3439.753487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.217447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.274049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3447.763926] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3456.374861] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3467.238521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3467.245754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.257580] LustreError: Skipped 1 previous similar message [ 3467.258421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3467.273793] Lustre: Skipped 1 previous similar message [ 3467.307548] Lustre: Skipped 2 previous similar messages [ 3469.950459] Lustre: server umount lustre-OST0000 complete [ 3477.475164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.492953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.595907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.607147] Lustre: Skipped 2 previous similar messages [ 3488.226083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3488.403927] Lustre: server umount lustre-MDT0000 complete [ 3492.568525] LustreError: 106303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627496 with bad export cookie 15510916928569821043 [ 3492.588975] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.595189] LustreError: 106303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3493.103971] Lustre: server umount lustre-MDT0001 complete [ 3507.615676] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3511.038737] Key type lgssc unregistered [ 3511.439277] LNet: 109860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3511.449713] LNetError: 109860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3511.480656] LNet: Removed LNI 192.168.204.106@tcp [ 3512.622695] Key type .llcrypt unregistered [ 3512.626214] Key type ._llcrypt unregistered [ 3541.228354] Key type ._llcrypt registered [ 3541.231578] Key type .llcrypt registered [ 3541.456617] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3543.166867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3543.473424] alg: No test for adler32 (adler32-zlib) [ 3544.699373] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3545.082862] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3546.800529] Key type lgssc registered [ 3548.521195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.993854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3558.014760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.503646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.297972] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3569.638765] LustreError: 111277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.681537] LustreError: 111277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3574.766316] LustreError: 111278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.844152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.474134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.885518] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3590.996817] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3598.674947] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3607.725715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.129395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.040385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3626.369642] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3629.619898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3637.281985] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3643.330852] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3650.021557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.031029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.050702] Lustre: Skipped 1 previous similar message [ 3650.067843] Lustre: Skipped 1 previous similar message [ 3655.142457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.618309] Lustre: server umount lustre-OST0000 complete [ 3663.332538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3663.342802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.374475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.386075] Lustre: Skipped 1 previous similar message [ 3665.378331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.397457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.424662] Lustre: Skipped 1 previous similar message [ 3666.925600] Lustre: server umount lustre-MDT0000 complete [ 3670.514559] LustreError: 111278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.548841] LustreError: 111278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3671.799945] LustreError: 111259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627675 with bad export cookie 15409665894112352016 [ 3671.823320] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.521937] Lustre: server umount lustre-MDT0001 complete [ 3686.079455] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3689.430959] Key type lgssc unregistered [ 3689.744864] LNet: 114923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3689.751053] LNetError: 114923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3689.764030] LNet: Removed LNI 192.168.204.106@tcp [ 3690.615913] Key type .llcrypt unregistered [ 3690.618368] Key type ._llcrypt unregistered [ 3712.037687] Key type ._llcrypt registered [ 3712.039707] Key type .llcrypt registered [ 3712.166680] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3713.329268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3713.655448] alg: No test for adler32 (adler32-zlib) [ 3714.604265] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3714.850144] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3716.536709] Key type lgssc registered [ 3717.763879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.150814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.196963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.759557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.091842] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3738.084855] LustreError: 116145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.110148] LustreError: 116145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3741.474968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.822783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3747.434292] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3755.758877] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3762.348521] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3770.634801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.375448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.416172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3780.343797] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3789.841448] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3802.085356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.087708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.091699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.097504] Lustre: Skipped 1 previous similar message [ 3806.029432] Lustre: server umount lustre-OST0000 complete [ 3812.333870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.347565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.360750] Lustre: Skipped 1 previous similar message [ 3812.385280] Lustre: Skipped 2 previous similar messages [ 3817.449296] LustreError: 116146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.486611] LustreError: 116146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3817.571937] Lustre: server umount lustre-MDT0000 complete [ 3822.315419] LustreError: 116125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627826 with bad export cookie 4393047896272873148 [ 3822.338469] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3822.347677] LustreError: 116125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3822.804824] Lustre: server umount lustre-MDT0001 complete [ 3838.800457] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3842.877697] Key type lgssc unregistered [ 3843.241793] LNet: 119684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.249759] LNetError: 119684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.267386] LNet: Removed LNI 192.168.204.106@tcp [ 3844.308608] Key type .llcrypt unregistered [ 3844.314057] Key type ._llcrypt unregistered [ 3874.238364] Key type ._llcrypt registered [ 3874.243202] Key type .llcrypt registered [ 3874.376429] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3875.612360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.675639] alg: No test for adler32 (adler32-zlib) [ 3876.938264] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3877.252702] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3879.016371] Key type lgssc registered [ 3880.721663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.433784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.455283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.132063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.852664] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3902.437588] LustreError: 121106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.469253] LustreError: 121106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3907.555129] LustreError: 121107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.452974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.838292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3914.710747] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3924.295808] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3930.764580] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.988148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.266566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.878546] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3957.183086] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3962.828880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3973.276592] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3977.951327] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3977.960079] Lustre: Skipped 2 previous similar messages [ 3981.438766] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3981.487593] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3982.767666] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3982.773187] Lustre: Skipped 2 previous similar messages [ 3987.030089] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3987.048588] Lustre: Skipped 1 previous similar message [ 3993.676474] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3993.695233] Lustre: Skipped 2 previous similar messages [ 4003.956412] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp [ 4009.685498] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.6/24@tcp in log lustre-MDT0000 [ 4009.700171] Lustre: Skipped 2 previous similar messages [ 4019.391870] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 4019.402643] Lustre: Skipped 2 previous similar messages [ 4029.416127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.423119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.435763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4029.443710] Lustre: Skipped 1 previous similar message [ 4029.455988] Lustre: Skipped 1 previous similar message [ 4034.533026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.661046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.669445] Lustre: Skipped 2 previous similar messages [ 4040.160141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4040.504576] Lustre: server umount lustre-OST0000 complete [ 4047.331671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4047.345875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.364042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4049.894107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.865758] Lustre: server umount lustre-MDT0000 complete [ 4055.015658] LustreError: 121106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.042890] LustreError: 121106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4056.185078] LustreError: 121088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628060 with bad export cookie 6236201891032321668 [ 4056.189937] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.196055] LustreError: 121088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4056.620398] Lustre: server umount lustre-MDT0001 complete [ 4070.641445] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4074.780463] Key type lgssc unregistered [ 4075.179764] LNet: 126223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4075.195883] LNetError: 126223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4075.213146] LNet: Removed LNI 192.168.204.106@tcp [ 4076.622613] Key type .llcrypt unregistered [ 4076.626119] Key type ._llcrypt unregistered [ 4110.024632] Key type ._llcrypt registered [ 4110.031369] Key type .llcrypt registered [ 4110.188311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4111.334954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4111.367178] alg: No test for adler32 (adler32-zlib) [ 4112.562122] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 4112.864956] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4114.592549] Key type lgssc registered [ 4115.641233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4124.640253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4124.675580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.351921] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4126.386948] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.6/24@tcp [ 4126.422946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.167744] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4136.935328] LustreError: 127787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.960147] LustreError: 127787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4142.063318] LustreError: 127786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.956273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.364104] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4143.410437] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 4143.445933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.642558] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4155.991131] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4161.955890] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4169.811411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.290872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.331994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4176.709611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4183.478562] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4198.891409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.898441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4198.912364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.931362] Lustre: Skipped 1 previous similar message [ 4204.004987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.036329] Lustre: Skipped 1 previous similar message [ 4204.965918] Lustre: server umount lustre-OST0000 complete [ 4210.150877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4210.156859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.164701] Lustre: Skipped 1 previous similar message [ 4210.168739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.245518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.246025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.942815] Lustre: server umount lustre-MDT0000 complete [ 4219.367388] LustreError: 127787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.400114] LustreError: 127787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4220.594883] LustreError: 127767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628224 with bad export cookie 17624983040941202560 [ 4220.612525] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.620617] LustreError: 127767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4221.780942] Lustre: server umount lustre-MDT0001 complete [ 4234.447578] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4238.235555] Key type lgssc unregistered [ 4238.622381] LNet: 131379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4238.628205] LNetError: 131379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4238.651776] LNet: Removed LNI 192.168.204.106@tcp [ 4240.004426] Key type .llcrypt unregistered [ 4240.013056] Key type ._llcrypt unregistered [ 4269.911354] Key type ._llcrypt registered [ 4269.914068] Key type .llcrypt registered [ 4270.082981] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4286.419288] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4287.212412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4287.237671] alg: No test for adler32 (adler32-zlib) [ 4288.383587] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 4288.689565] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4290.416298] Key type lgssc registered [ 4292.114855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4331.932422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4331.986211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.310189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4333.352881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4333.441248] Lustre: lustre-MDT0000: new disk, initializing [ 4333.557401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.588141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4337.718861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4347.235344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.301761] Lustre: 134780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4347.345781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4347.346933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4347.359695] Lustre: Skipped 1 previous similar message [ 4347.425455] Lustre: lustre-MDT0001: new disk, initializing [ 4347.501731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4347.545133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4347.555404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4352.757362] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4362.378732] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4368.654520] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4376.480850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.941151] Lustre: lustre-OST0000: new disk, initializing [ 4376.950674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4376.965554] Lustre: 136749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4377.136646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.652471] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4385.995403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4386.018146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4386.251194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4397.474711] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4403.870082] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4404.288261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4411.134377] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4411.395716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4417.016126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.026540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4417.045418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4417.061333] Lustre: Skipped 1 previous similar message [ 4419.853223] Lustre: server umount lustre-OST0000 complete [ 4427.256263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.257100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.304171] Lustre: Skipped 1 previous similar message [ 4427.326241] Lustre: Skipped 2 previous similar messages [ 4430.838972] Lustre: server umount lustre-MDT0000 complete [ 4437.375380] LustreError: 134771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628441 with bad export cookie 7619347591881768385 [ 4437.383523] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.393358] LustreError: 134771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4437.763240] Lustre: server umount lustre-MDT0001 complete [ 4453.324972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.328644] LustreError: 138967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.344661] LustreError: 138967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4454.394562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4459.308785] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4459.489821] LustreError: 138968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.611459] LustreError: 138967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.105461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.437931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4472.570210] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4481.520287] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4488.848779] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4497.176503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4497.687149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.697385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4514.425090] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4519.921131] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4520.242136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4524.684311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4524.879355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4534.325881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.522407] Lustre: lustre-OST0001: new disk, initializing [ 4534.525683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4534.532134] Lustre: 142339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4534.665317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.058745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4540.070847] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4540.206852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4542.607215] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4552.235657] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4552.510433] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4558.155423] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4558.460514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4564.463298] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c94a400800.ost_server_uuid 50 [ 4566.207984] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c94a400800.ost_server_uuid in IDLE state after 0 sec [ 4581.353904] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.358886] LustreError: Skipped 1 previous similar message [ 4581.364957] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.379719] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4581.384111] Lustre: Skipped 1 previous similar message [ 4583.396634] Lustre: server umount lustre-OST0001 complete [ 4586.467661] LustreError: 140791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.510288] LustreError: 140791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4590.050399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4590.063684] LustreError: Skipped 1 previous similar message [ 4590.074985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.097498] Lustre: Skipped 1 previous similar message [ 4590.106200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4591.589543] LustreError: 140792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.616126] LustreError: 140792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4593.771593] Lustre: server umount lustre-OST0000 complete [ 4600.804946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4600.812720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.822813] Lustre: Skipped 1 previous similar message [ 4600.828535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4600.836196] Lustre: Skipped 2 previous similar messages [ 4612.064472] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4612.325537] Lustre: server umount lustre-MDT0000 complete [ 4616.167223] LustreError: 139962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.184539] LustreError: 139962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4616.609835] LustreError: 138949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628620 with bad export cookie 7619347591881769260 [ 4616.619254] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.621958] LustreError: 138949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4617.083474] Lustre: server umount lustre-MDT0001 complete [ 4637.784994] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4641.221581] Key type lgssc unregistered [ 4641.637684] LNet: 145347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.648640] LNetError: 145347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4641.672951] LNet: Removed LNI 192.168.204.106@tcp [ 4642.641708] Key type .llcrypt unregistered [ 4642.645492] Key type ._llcrypt unregistered [ 4685.063337] Key type ._llcrypt registered [ 4685.066271] Key type .llcrypt registered [ 4685.253896] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4703.303532] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4704.076360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4704.134141] alg: No test for adler32 (adler32-zlib) [ 4705.184604] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 4705.515737] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4707.248164] Key type lgssc registered [ 4708.613860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4746.969592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.020939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.254754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4748.305141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4748.372433] Lustre: lustre-MDT0000: new disk, initializing [ 4748.520041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4748.538609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4752.570450] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4765.490394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.692551] Lustre: 149506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4765.824511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4765.839036] Lustre: Skipped 1 previous similar message [ 4766.031723] Lustre: lustre-MDT0001: new disk, initializing [ 4766.212527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4766.274853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4766.294653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4772.316876] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4782.756364] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4789.344034] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4796.610172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4796.918610] Lustre: lustre-OST0000: new disk, initializing [ 4796.927812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4796.940570] Lustre: 151471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4797.052610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.709836] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4803.128087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4803.136276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4803.224285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4811.661740] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4817.085844] Lustre: Failing over lustre-OST0000 [ 4817.331817] Lustre: server umount lustre-OST0000 complete [ 4834.208869] Lustre: 147730:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628822/real 1787628822] req@000000002b84b61e x1874464561202816/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787628838 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4834.249186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.471761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4837.787408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.812022] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4839.392132] Lustre: 147731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628827/real 1787628827] req@000000006e9df9aa x1874464561204736/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787628843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4839.435961] Lustre: 147731:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4843.803577] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4844.901234] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4845.123518] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4845.123547] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4845.921475] Lustre: 147731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628833/real 1787628833] req@00000000a3667cab x1874464561205760/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787628849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4845.957762] Lustre: 147731:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4847.142638] Lustre: Failing over lustre-MDT0000 [ 4847.403703] Lustre: server umount lustre-MDT0000 complete [ 4848.099314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4848.106739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.118831] Lustre: Skipped 1 previous similar message [ 4850.147039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.168844] Lustre: Skipped 1 previous similar message [ 4855.278402] LustreError: 149511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.305015] LustreError: 149511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4860.389574] LustreError: 149517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.419111] LustreError: 149517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4865.506546] LustreError: 149516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.541102] LustreError: 149516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4865.705227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.871378] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.160666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.209985] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4866.749617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4870.122748] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4871.657437] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4871.662339] Lustre: Skipped 1 previous similar message [ 4871.684094] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4871.727282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4881.385424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.392690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.406235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.880497] Lustre: server umount lustre-OST0000 complete [ 4892.130522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4892.132797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.141905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.158834] Lustre: Skipped 2 previous similar messages [ 4892.169405] Lustre: Skipped 1 previous similar message [ 4894.744979] Lustre: server umount lustre-MDT0000 complete [ 4897.250067] LustreError: 149516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.268816] LustreError: 149516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4898.936146] LustreError: 149497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628902 with bad export cookie 7793373577427061234 [ 4898.951577] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.969066] LustreError: 149497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4899.007434] LustreError: 147730:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 4899.513163] Lustre: server umount lustre-MDT0001 complete [ 4910.508591] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4913.314987] Key type lgssc unregistered [ 4913.642776] LNet: 155133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4913.654607] LNetError: 155133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4913.672821] LNet: Removed LNI 192.168.204.106@tcp [ 4914.527794] Key type .llcrypt unregistered [ 4914.530258] Key type ._llcrypt unregistered [ 4949.069121] Key type ._llcrypt registered [ 4949.079656] Key type .llcrypt registered [ 4949.202993] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4950.038947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4950.060838] alg: No test for adler32 (adler32-zlib) [ 4951.081724] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 4951.265964] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4952.984138] Key type lgssc registered [ 4954.164514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4960.800136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4960.817095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.313776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.583720] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4972.517405] LustreError: 156670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.535375] LustreError: 156670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4975.187215] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4977.636928] LustreError: 156671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4979.128860] LustreError: 156700:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4979.153246] LustreError: 156700:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4979.160448] LustreError: 156700:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4979.570304] Lustre: server umount lustre-MDT0000 complete [ 4995.657269] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5005.254928] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5042.403091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.778044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5042.808403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5042.892247] Lustre: lustre-MDT0000: new disk, initializing [ 5042.964721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.980810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5046.941426] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5057.489583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.582467] Lustre: 160002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.622278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5057.624408] Lustre: Skipped 1 previous similar message [ 5057.670703] Lustre: lustre-MDT0001: new disk, initializing [ 5057.736209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.761389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5057.770198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5062.282990] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5072.440749] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5077.398632] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5084.820086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5085.088403] Lustre: lustre-OST0000: new disk, initializing [ 5085.101692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5085.110276] Lustre: 161972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5085.209138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.418971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5090.434827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5090.624620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5092.587853] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5104.566360] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5109.762080] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5112.117631] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:38:33 (1787629113) [ 5375.456801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5375.466676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.479939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.514340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.517043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.535153] Lustre: Skipped 1 previous similar message [ 5377.545517] Lustre: Skipped 2 previous similar messages [ 5380.916657] Lustre: server umount lustre-MDT0000 complete [ 5382.626061] LustreError: 161262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.649556] LustreError: 161262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5384.397813] LustreError: 159994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787629388 with bad export cookie 11339514136286922384 [ 5384.403210] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.414166] LustreError: 159994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5387.748707] LustreError: 160010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.751841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.783661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5389.792701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5390.842212] Lustre: server umount lustre-MDT0001 complete [ 5401.596506] Lustre: server umount lustre-OST0000 complete [ 5412.112223] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5422.462198] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5457.449376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.667453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5457.703884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5457.803891] Lustre: lustre-MDT0000: new disk, initializing [ 5457.869269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.884797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5461.453541] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5471.877953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.085864] Lustre: 166636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5472.185496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5472.191512] Lustre: Skipped 1 previous similar message [ 5472.375709] Lustre: lustre-MDT0001: new disk, initializing [ 5472.471895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.516595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5472.544797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5478.446497] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5488.791341] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5494.170925] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5501.201812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5501.513714] Lustre: lustre-OST0000: new disk, initializing [ 5501.517442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5501.524705] Lustre: 168604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5501.626707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.000148] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5510.248372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5510.267389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5510.424318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5517.626160] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5521.981483] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5522.155994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5526.073835] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5526.308740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5531.111471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.116687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.117622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5531.125933] Lustre: Skipped 1 previous similar message [ 5531.140773] Lustre: Skipped 1 previous similar message [ 5541.349651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5541.361349] Lustre: Skipped 3 previous similar messages [ 5542.368148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5542.550227] Lustre: server umount lustre-OST0000 complete [ 5546.470456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.289144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5560.500898] Lustre: server umount lustre-MDT0000 complete [ 5561.826351] LustreError: 168611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5561.852517] LustreError: 168611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5563.698117] LustreError: 166629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787629567 with bad export cookie 11339514136287081228 [ 5563.703727] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.709944] LustreError: 166629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5564.044771] Lustre: server umount lustre-MDT0001 complete [ 5572.276634] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:46:13 (1787629573) [ 5588.319365] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5597.255317] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5630.258844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.431828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5630.466421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5630.553570] Lustre: lustre-MDT0000: new disk, initializing [ 5630.621541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.641202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5635.650420] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5647.306967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.538214] Lustre: 173368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5647.865559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5653.125640] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5661.133681] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5665.238437] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5671.435212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5671.716416] Lustre: lustre-OST0000: new disk, initializing [ 5671.724331] Lustre: Skipped 1 previous similar message [ 5671.730400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5671.739294] Lustre: Skipped 2 previous similar messages [ 5671.748771] Lustre: 175346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5677.611554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5677.620941] Lustre: Skipped 1 previous similar message [ 5677.629114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5677.804647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5677.822546] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5687.091105] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5693.411474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.414140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5693.419681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.419997] Lustre: Skipped 6 previous similar messages [ 5693.422419] Lustre: Skipped 2 previous similar messages [ 5694.694747] Lustre: server umount lustre-OST0000 complete [ 5712.352132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5712.446287] Lustre: server umount lustre-MDT0000 complete [ 5713.894378] LustreError: 173377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.922562] LustreError: 173377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5715.359822] LustreError: 173363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787629719 with bad export cookie 11339514136287082110 [ 5715.361201] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.371215] LustreError: 173363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5715.631585] Lustre: server umount lustre-MDT0001 complete [ 5722.121940] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:48:43 (1787629723) [ 5739.968988] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5750.206777] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5784.751544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5784.970459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5785.017170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5785.090374] Lustre: lustre-MDT0000: new disk, initializing [ 5785.231861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.235137] Lustre: Skipped 2 previous similar messages [ 5785.264783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5789.210630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5799.004483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.221179] Lustre: 179665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5799.566870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5804.102282] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5812.217896] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5816.219926] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5821.142587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5821.316308] Lustre: 181642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5827.447208] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5830.877624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5838.443781] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5851.620731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.621106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5851.623037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5851.623044] Lustre: Skipped 7 previous similar messages [ 5851.632957] Lustre: Skipped 3 previous similar messages [ 5851.677714] LustreError: Skipped 1 previous similar message [ 5853.340375] Lustre: server umount lustre-OST0000 complete [ 5860.833360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5863.412311] Lustre: server umount lustre-MDT0000 complete [ 5866.589428] LustreError: 179660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787629870 with bad export cookie 11339514136287083195 [ 5866.593794] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.597395] LustreError: 179660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5866.937475] Lustre: server umount lustre-MDT0001 complete [ 5874.691590] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5877.059201] Key type lgssc unregistered [ 5877.300747] LNet: 183500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5877.304289] LNetError: 183500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5878.375595] LNet: Removed LNI 192.168.204.106@tcp [ 5878.937609] Key type .llcrypt unregistered [ 5878.940520] Key type ._llcrypt unregistered [ 5902.735751] Key type ._llcrypt registered [ 5902.737994] Key type .llcrypt registered [ 5902.843749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5903.822434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5903.865993] alg: No test for adler32 (adler32-zlib) [ 5904.886705] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 5905.057520] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5906.680984] Key type lgssc registered [ 5907.947689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5915.101312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5915.130243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.639162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.522938] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5926.881604] LustreError: 184935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.897066] LustreError: 184935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5927.414321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.735384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.518882] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5936.987570] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5941.908593] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5948.793451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.090652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.837569] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5961.398905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5961.864892] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5982.177864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5982.185704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.187519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5982.194780] Lustre: Skipped 1 previous similar message [ 5982.213451] Lustre: Skipped 1 previous similar message [ 5985.301838] Lustre: server umount lustre-OST0000 complete [ 5989.348807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.365657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.376619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.418518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.421552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.441128] Lustre: Skipped 1 previous similar message [ 5997.543401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5997.555897] Lustre: Skipped 1 previous similar message [ 6002.657786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.169936] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6003.375344] Lustre: server umount lustre-MDT0000 complete [ 6007.029237] LustreError: 184916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630010 with bad export cookie 4009412691643618415 [ 6007.031489] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6007.036760] LustreError: 184916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6007.487651] Lustre: server umount lustre-MDT0001 complete [ 6017.712777] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6021.498756] Key type lgssc unregistered [ 6021.761340] LNet: 188497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6021.769292] LNetError: 188497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6021.795720] LNet: Removed LNI 192.168.204.106@tcp [ 6023.060803] Key type .llcrypt unregistered [ 6023.067625] Key type ._llcrypt unregistered [ 6048.532649] Key type ._llcrypt registered [ 6048.536896] Key type .llcrypt registered [ 6048.660422] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6049.786179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6049.802225] alg: No test for adler32 (adler32-zlib) [ 6050.855287] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6051.123701] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6052.824204] Key type lgssc registered [ 6053.711978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6059.616905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.635977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.988885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.404465] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6070.255223] LustreError: 189919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.292331] LustreError: 189919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6072.403248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.695510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.865711] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6085.094518] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6090.670250] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.745612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6097.130828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.580436] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6106.429435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6108.727894] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6116.465702] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 6116.841017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6116.844257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.860077] Lustre: Skipped 1 previous similar message [ 6121.545130] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 6121.549257] Lustre: Skipped 2 previous similar messages [ 6121.950271] Lustre: server umount lustre-OST0000 complete [ 6130.326260] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93c949d23000.ost_server_uuid 50 [ 6131.934245] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6146.537050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.577083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.596440] Lustre: Skipped 1 previous similar message [ 6151.652301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.657404] Lustre: Skipped 1 previous similar message [ 6152.629196] Lustre: server umount lustre-MDT0000 complete [ 6156.770880] LustreError: 192854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.773199] LustreError: 189901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630160 with bad export cookie 10566568453234859428 [ 6156.779035] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.825208] LustreError: 189901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6157.425888] Lustre: server umount lustre-MDT0001 complete [ 6168.258362] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:56:09 (1787630169) [ 6175.445449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.886299] LustreError: 193836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.050597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.150861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6181.353959] LustreError: 193836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.379726] LustreError: 193836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6185.454631] LustreError: 193835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6190.448677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.993059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6196.149831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6204.868970] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.013435] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.496930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.736686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.850143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6222.241973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6228.608302] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6234.195757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.440729] Lustre: lustre-OST0001: new disk, initializing [ 6234.449759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6234.458208] Lustre: 196676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6234.578633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6239.859858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6239.866853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6240.003846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6240.797600] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6250.957546] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6255.588843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.598635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.611064] Lustre: Skipped 2 previous similar messages [ 6255.626282] Lustre: Skipped 1 previous similar message [ 6259.633340] Lustre: server umount lustre-OST0000 complete [ 6260.710179] LustreError: 195657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.728815] LustreError: 195657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6264.793493] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6264.963337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6268.466320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6268.710314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6277.201099] LustreError: 195655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.227712] LustreError: 195655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6286.309959] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6286.312708] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6286.325899] LustreError: Skipped 1 previous similar message [ 6286.327253] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6286.338699] Lustre: Skipped 1 previous similar message [ 6286.353627] Lustre: Skipped 2 previous similar messages [ 6288.501510] Lustre: server umount lustre-OST0001 complete [ 6293.474684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6293.489967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.373827] Lustre: server umount lustre-MDT0000 complete [ 6302.428689] LustreError: 193816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630306 with bad export cookie 10566568453234860506 [ 6302.446431] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.457734] LustreError: 193816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6302.963624] Lustre: server umount lustre-MDT0001 complete [ 6330.928314] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:58:52 (1787630332) [ 6338.116927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.489601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6338.523878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6338.720689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.299047] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6351.729948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.811698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6351.846795] Lustre: 200119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.525960] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6367.236634] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6377.325892] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6386.360667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.494304] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6391.989493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6393.646540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6401.018798] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.646889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6407.734906] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6407.950919] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6407.960436] Lustre: Skipped 2 previous similar messages [ 6412.824984] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6414.526287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6419.649865] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6423.011911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.013941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.025818] LustreError: Skipped 1 previous similar message [ 6423.042036] Lustre: Skipped 1 previous similar message [ 6423.050669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.079457] Lustre: Skipped 4 previous similar messages [ 6428.541816] Lustre: server umount lustre-OST0000 complete [ 6430.202894] LustreError: 202240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.217275] LustreError: 202240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6445.550629] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.552402] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.575678] Lustre: Skipped 2 previous similar messages [ 6450.782728] Lustre: server umount lustre-OST0001 complete [ 6454.757517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6454.771512] LustreError: Skipped 1 previous similar message [ 6455.777544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6455.786152] Lustre: Skipped 9 previous similar messages [ 6460.770954] Lustre: server umount lustre-MDT0000 complete [ 6464.503286] LustreError: 200110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630468 with bad export cookie 10566568453234861913 [ 6464.504606] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6464.513794] LustreError: 200110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6464.783593] Lustre: server umount lustre-MDT0001 complete [ 6491.918497] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:01:33 (1787630493) [ 6509.465075] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6525.306890] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6570.839227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.207350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6571.227856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6571.324773] Lustre: lustre-MDT0000: new disk, initializing [ 6571.451593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.489995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6577.927650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6590.606430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.743839] Lustre: 208257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6590.875264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6590.883744] Lustre: Skipped 1 previous similar message [ 6590.944614] Lustre: lustre-MDT0001: new disk, initializing [ 6591.155165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6591.174462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6596.369242] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6609.426088] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6619.707550] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6631.027963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.625546] Lustre: lustre-OST0000: new disk, initializing [ 6631.634559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6631.651940] Lustre: 210227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.726710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6638.735228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6638.833657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6638.927647] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6649.224444] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6654.633628] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6654.899221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6660.292515] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6660.572656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.677057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6664.677311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.682248] LustreError: Skipped 1 previous similar message [ 6664.686592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.688368] Lustre: Skipped 3 previous similar messages [ 6664.699439] Lustre: Skipped 1 previous similar message [ 6668.831751] Lustre: server umount lustre-OST0000 complete [ 6678.870167] Lustre: server umount lustre-MDT0000 complete [ 6680.037865] LustreError: 208263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.049914] LustreError: 208263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6683.058977] LustreError: 208248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630686 with bad export cookie 10566568453234863698 [ 6683.086316] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.088228] LustreError: 208248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6683.632994] Lustre: server umount lustre-MDT0001 complete [ 6694.264095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.841290] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6710.862952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.713185] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6724.565496] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6729.926807] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6741.363408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.889369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6762.079915] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.767831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6767.961479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6773.540598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6773.885360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6780.387954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.391940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6780.404775] Lustre: Skipped 3 previous similar messages [ 6780.418819] LustreError: Skipped 1 previous similar message [ 6782.818856] Lustre: server umount lustre-OST0000 complete [ 6790.297054] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6797.366468] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6829.405722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6829.903221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6829.912705] Lustre: Skipped 5 previous similar messages [ 6837.387472] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6845.245314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6845.281749] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6845.283686] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6845.324394] Lustre: Skipped 1 previous similar message [ 6846.245480] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6848.595095] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6855.650026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6855.653438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6855.669424] LustreError: Skipped 1 previous similar message [ 6855.692651] Lustre: Skipped 6 previous similar messages [ 6860.007487] Lustre: server umount lustre-OST0000 complete [ 6870.579508] Lustre: server umount lustre-MDT0000 complete [ 6871.009833] LustreError: 212437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.034865] LustreError: 212437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6875.306762] LustreError: 212423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787630879 with bad export cookie 10566568453234864580 [ 6875.328356] LustreError: 212423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6875.332019] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6876.075984] Lustre: server umount lustre-MDT0001 complete [ 6887.559922] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:08:08 (1787630888) [ 6896.945492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.418517] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6913.944883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.916223] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6926.924848] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.707681] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6943.961396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.719152] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6961.346770] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.750091] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6968.013127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6973.801913] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6974.134269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6981.852562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.156960] Lustre: lustre-OST0001: new disk, initializing [ 6982.159761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6982.164031] Lustre: 221796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6990.754588] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6992.558045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6992.580095] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6992.719865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7001.566846] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7007.276736] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7007.573780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7012.440145] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7012.654448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7018.476584] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.493503] Lustre: Skipped 4 previous similar messages [ 7018.518614] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7018.530042] LustreError: Skipped 1 previous similar message [ 7021.378419] Lustre: server umount lustre-OST0001 complete [ 7029.358308] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7029.586805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7034.948174] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7035.120656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7065.158745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.707199] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7070.723543] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7070.724452] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7070.757268] Lustre: Skipped 1 previous similar message [ 7072.481970] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7080.495710] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7082.022474] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7090.336464] Lustre: server umount lustre-OST0001 complete [ 7111.652855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.668877] Lustre: Skipped 10 previous similar messages [ 7114.229451] Lustre: server umount lustre-OST0000 complete [ 7124.750533] Lustre: server umount lustre-MDT0000 complete [ 7127.016972] LustreError: 218419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.035506] LustreError: 218419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7129.657680] LustreError: 218399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787631133 with bad export cookie 10566568453234865546 [ 7129.700365] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.343473] Lustre: server umount lustre-MDT0001 complete [ 7165.912854] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:12:47 (1787631167) [ 7173.688166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.149752] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7174.184606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7179.545966] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7189.581408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.755616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7189.795614] Lustre: 227042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7195.275540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7203.490863] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7210.419272] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7218.677118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.945236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7226.442219] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7235.549397] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.919307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.965122] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7253.487421] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7263.183889] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7265.360741] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7270.981214] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7271.271365] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7276.201749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7276.465984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7283.359934] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c948828000.ost_server_uuid 50 [ 7285.219227] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c948828000.ost_server_uuid in IDLE state after 0 sec [ 7286.462936] Lustre: Permanently deactivating lustre-OST0001 [ 7286.485710] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7291.650801] Lustre: Permanently reactivating lustre-OST0001 [ 7291.668981] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7291.673607] Lustre: Skipped 2 previous similar messages [ 7297.002131] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7297.010816] LustreError: Skipped 5 previous similar messages [ 7297.015457] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.029568] Lustre: Skipped 6 previous similar messages [ 7302.877575] Lustre: server umount lustre-OST0001 complete [ 7322.082262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7322.438449] Lustre: server umount lustre-OST0000 complete [ 7333.930655] Lustre: server umount lustre-MDT0000 complete [ 7337.746832] LustreError: 231450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787631341 with bad export cookie 10566568453234866834 [ 7337.749082] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.765487] LustreError: 231450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7338.130200] Lustre: server umount lustre-MDT0001 complete [ 7366.398768] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:16:07 (1787631367) [ 7375.619741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.056116] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7376.106797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7376.119306] Lustre: Skipped 2 previous similar messages [ 7376.432634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.446447] Lustre: Skipped 9 previous similar messages [ 7382.005650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7391.307243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.423174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7391.470281] Lustre: 233738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7396.764319] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7405.417876] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7411.441117] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7418.346240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.520812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7418.538868] Lustre: Permanently deactivating lustre-OST0000 [ 7418.553475] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7424.922368] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7425.051277] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7430.840409] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7437.621083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7437.730520] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7444.682837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7445.638749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7454.184235] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7455.826740] Lustre: Permanently reactivating lustre-OST0000 [ 7455.836575] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7455.840176] Lustre: Skipped 2 previous similar messages [ 7463.985320] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7463.995510] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 00000000b7499cc9, cur 1787631467 deadline 1787631459 last 1787631429 [ 7464.041318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7479.072768] Lustre: server umount lustre-OST0001 complete [ 7489.386650] Lustre: server umount lustre-OST0000 complete [ 7500.993537] Lustre: server umount lustre-MDT0000 complete [ 7504.465297] LustreError: 233730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787631508 with bad export cookie 10566568453234868507 [ 7504.471579] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7504.478332] LustreError: 233730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7504.857277] Lustre: server umount lustre-MDT0001 complete [ 7516.100256] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7518.433916] Key type lgssc unregistered [ 7518.662647] LNet: 238627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7518.672694] LNetError: 238627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7518.686180] LNet: Removed LNI 192.168.204.106@tcp [ 7519.499803] Key type .llcrypt unregistered [ 7519.508927] Key type ._llcrypt unregistered [ 7544.825469] Key type ._llcrypt registered [ 7544.831432] Key type .llcrypt registered [ 7545.008936] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7546.324696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7546.401018] alg: No test for adler32 (adler32-zlib) [ 7547.730902] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7548.120038] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7549.864241] Key type lgssc registered [ 7550.944793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7561.202654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7561.234163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.790585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.221595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7572.974308] LustreError: 240094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.009031] LustreError: 240094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7577.355593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.534683] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7577.570401] Lustre: 240080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7577.583668] Lustre: Permanently deactivating lustre-MDT0001 [ 7577.596475] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7578.012463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7579.806292] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7579.821395] LustreError: 240124:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7579.845093] LustreError: 240124:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7579.859504] LustreError: 240124:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7583.194456] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7590.383563] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7595.888382] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7604.616662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7605.216304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.273777] LustreError: 241935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.303314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7612.409130] LustreError: 241937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.279909] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7617.508824] LustreError: 241935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7622.013984] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7622.634498] LustreError: 241936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7622.661840] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7622.666961] LustreError: 241936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7629.918895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.210713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.443052] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7635.470607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7637.058711] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7644.109620] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7647.791759] Lustre: Permanently reactivating lustre-MDT0001 [ 7647.827248] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7647.835506] Lustre: Skipped 2 previous similar messages [ 7652.463926] Lustre: lustre-MDT0001: Client dd8de86a-7be1-4b14-b82b-249972d1d31d (at 192.168.204.6@tcp) reconnecting [ 7659.285270] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7661.327931] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7667.701645] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7668.021663] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7673.820457] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7674.170766] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7676.926209] Lustre: Permanently deactivating lustre-MDT0001 [ 7676.944513] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7676.949703] Lustre: Skipped 2 previous similar messages [ 7682.655885] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7691.751710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.753785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7691.769434] Lustre: Skipped 2 previous similar messages [ 7691.781686] Lustre: Skipped 2 previous similar messages [ 7693.772966] Lustre: server umount lustre-MDT0000 complete [ 7696.878179] LustreError: 240094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.896281] LustreError: 240094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7697.544314] LustreError: 240073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787631701 with bad export cookie 10544720263133999847 [ 7697.550918] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7697.555344] LustreError: 240073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7697.858739] Lustre: server umount lustre-MDT0001 complete [ 7703.696695] Lustre: server umount lustre-OST0000 complete [ 7714.908442] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:21:56 (1787631716) [ 7725.142170] Lustre: server umount lustre-OST0001 complete [ 7732.394348] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7742.373716] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7776.949198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.086880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7777.093281] Lustre: Skipped 2 previous similar messages [ 7777.112256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7777.198714] Lustre: lustre-MDT0000: new disk, initializing [ 7777.278687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.302659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7781.464956] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7791.853755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.962362] Lustre: 248353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7792.007308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7792.011020] Lustre: Skipped 1 previous similar message [ 7792.066626] Lustre: lustre-MDT0001: new disk, initializing [ 7792.152115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7792.205523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7792.219104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7795.926585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7805.114547] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7810.446478] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7816.481464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.743037] Lustre: lustre-OST0000: new disk, initializing [ 7816.747251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7816.754091] Lustre: 250319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7816.858261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.252844] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7823.413908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7823.426275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7823.555527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7832.549229] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7834.572407] LustreError: 248360:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7843.348854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.541762] Lustre: lustre-OST0001: new disk, initializing [ 7843.548862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7843.561899] Lustre: 251433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7844.576101] LustreError: 248360:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7844.606990] LustreError: 248359:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7849.563986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7849.572709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7849.639239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7849.964407] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7854.704104] LustreError: 248359:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7859.042518] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7860.368149] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7862.374810] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 7865.320743] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7865.325183] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7865.330975] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7865.381436] Lustre: Skipped 2 previous similar messages [ 7867.490257] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 7867.495943] Lustre: Skipped 1 previous similar message [ 7868.324923] Lustre: server umount lustre-OST0001 complete [ 7870.434073] LustreError: 250303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.452322] LustreError: 250303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7885.281625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7885.293070] LustreError: Skipped 1 previous similar message [ 7885.302955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.315863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7885.320437] Lustre: Skipped 1 previous similar message [ 7896.047436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7896.076166] Lustre: Skipped 3 previous similar messages [ 7898.080132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7898.264938] Lustre: server umount lustre-OST0000 complete [ 7904.748107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7904.759518] LustreError: Skipped 1 previous similar message [ 7904.762272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.772451] Lustre: Skipped 1 previous similar message [ 7909.110550] Lustre: server umount lustre-MDT0000 complete [ 7911.396388] LustreError: 248358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.414402] LustreError: 248358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7913.129205] LustreError: 251432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787631917 with bad export cookie 10544720263134002661 [ 7913.136905] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.144070] LustreError: 251432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7919.931685] Lustre: server umount lustre-MDT0001 complete [ 7930.968372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7933.879407] Key type lgssc unregistered [ 7934.139688] LNet: 253471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7934.148529] LNetError: 253471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7934.170914] LNet: Removed LNI 192.168.204.106@tcp [ 7935.155286] Key type .llcrypt unregistered [ 7935.158243] Key type ._llcrypt unregistered [ 7974.794679] Key type ._llcrypt registered [ 7974.796669] Key type .llcrypt registered [ 7974.988037] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7976.191837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7976.322565] alg: No test for adler32 (adler32-zlib) [ 7977.413405] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7977.706388] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7979.448690] Key type lgssc registered [ 7980.787056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.249478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7988.262041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.422271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7989.441977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7989.643919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.280056] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8002.480129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.625410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8002.677817] Lustre: 255682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8003.029564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8007.249355] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8015.054989] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.620708] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8026.831351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.022828] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8027.284888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.464923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8034.281315] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8041.964587] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8094.691833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.697841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.730671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8095.206553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.217853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8099.546415] Lustre: server umount lustre-OST0000 complete [ 8104.468471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8117.690150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8118.189375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.238575] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8120.260147] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8124.271492] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8131.343839] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8135.588711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8135.620499] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8171.491133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.498235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.504217] LustreError: Skipped 1 previous similar message [ 8171.506022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.506029] Lustre: Skipped 2 previous similar messages [ 8171.544315] Lustre: Skipped 1 previous similar message [ 8176.431244] Lustre: server umount lustre-OST0000 complete [ 8181.729745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.745062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.745827] Lustre: Skipped 1 previous similar message [ 8181.767280] Lustre: Skipped 2 previous similar messages [ 8186.486191] Lustre: server umount lustre-MDT0000 complete [ 8190.188798] LustreError: 255674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632194 with bad export cookie 6551652569533746697 [ 8190.190055] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8190.198927] LustreError: 255674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8190.592072] Lustre: server umount lustre-MDT0001 complete [ 8202.299852] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8205.761419] Key type lgssc unregistered [ 8206.157616] LNet: 261114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8206.166839] LNetError: 261114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8206.188201] LNet: Removed LNI 192.168.204.106@tcp [ 8207.049290] Key type .llcrypt unregistered [ 8207.057526] Key type ._llcrypt unregistered [ 8230.591695] Key type ._llcrypt registered [ 8230.595149] Key type .llcrypt registered [ 8230.692657] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8231.628634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8231.819111] alg: No test for adler32 (adler32-zlib) [ 8232.949102] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8233.334719] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8235.096424] Key type lgssc registered [ 8236.656907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8245.006631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8245.025446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.542500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.156050] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8257.003204] LustreError: 262531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8257.030154] LustreError: 262531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8259.478355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.868629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8264.827791] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8273.142954] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.784362] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.434291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.885398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8292.016043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8293.829768] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8300.913927] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8330.212916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.248625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8335.333632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8335.342888] Lustre: Skipped 1 previous similar message [ 8336.363920] Lustre: server umount lustre-OST0000 complete [ 8340.450392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.482318] Lustre: Skipped 1 previous similar message [ 8340.491415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.496849] Lustre: Skipped 1 previous similar message [ 8345.573759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.580262] Lustre: Skipped 1 previous similar message [ 8350.690908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8350.706670] Lustre: Skipped 1 previous similar message [ 8354.273264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8354.471974] Lustre: server umount lustre-MDT0000 complete [ 8355.810724] LustreError: 262530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.835797] LustreError: 262530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8357.836853] LustreError: 262510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632361 with bad export cookie 11624837021355871723 [ 8357.845429] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8357.855066] LustreError: 262510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8358.284678] Lustre: server umount lustre-MDT0001 complete [ 8367.353677] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8369.899643] Key type lgssc unregistered [ 8370.142755] LNet: 266843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8370.151368] LNetError: 266843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8370.177056] LNet: Removed LNI 192.168.204.106@tcp [ 8371.103219] Key type .llcrypt unregistered [ 8371.105782] Key type ._llcrypt unregistered [ 8388.761583] Key type ._llcrypt registered [ 8388.764841] Key type .llcrypt registered [ 8388.870360] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8389.901735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8389.927094] alg: No test for adler32 (adler32-zlib) [ 8390.874087] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8391.038689] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8392.672224] Key type lgssc registered [ 8393.992676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8401.243847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8401.301675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.777184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.227559] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8412.141709] LustreError: 268068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.159921] LustreError: 268068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8414.928807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8415.384384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8419.464968] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8426.027527] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8431.308562] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8438.524213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.086059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.926611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8447.282455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8453.134267] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8466.920970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8466.932957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.955246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.940409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8467.957160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.957202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.992816] Lustre: Skipped 1 previous similar message [ 8469.454142] Lustre: server umount lustre-OST0000 complete [ 8473.057564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8473.072857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8473.083271] Lustre: Skipped 1 previous similar message [ 8478.179074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8478.187277] Lustre: Skipped 1 previous similar message [ 8483.300481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8483.311844] Lustre: Skipped 1 previous similar message [ 8487.392571] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8487.667840] Lustre: server umount lustre-MDT0000 complete [ 8488.417058] LustreError: 268067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.441819] LustreError: 268067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8491.438495] LustreError: 268048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632495 with bad export cookie 13970886206746042360 [ 8491.444885] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8491.449086] LustreError: 268048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8492.019531] Lustre: server umount lustre-MDT0001 complete [ 8502.283498] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8505.066631] Key type lgssc unregistered [ 8505.360827] LNet: 271747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8505.376428] LNetError: 271747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8505.413803] LNet: Removed LNI 192.168.204.106@tcp [ 8506.223270] Key type .llcrypt unregistered [ 8506.226024] Key type ._llcrypt unregistered [ 8529.547744] Key type ._llcrypt registered [ 8529.551554] Key type .llcrypt registered [ 8529.672716] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8531.068352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8531.136899] alg: No test for adler32 (adler32-zlib) [ 8532.254857] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8532.559404] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8534.280233] Key type lgssc registered [ 8535.616182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8544.066490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8544.086045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.512183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.304865] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8556.006888] LustreError: 273165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.043583] LustreError: 273165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8560.551132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.938657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8565.517586] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8572.392441] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8578.236319] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8584.725562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.194329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.337533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8591.793784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8600.382209] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8626.146197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.154953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.162827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.172542] Lustre: Skipped 1 previous similar message [ 8630.618542] Lustre: server umount lustre-OST0000 complete [ 8636.387750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.403970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8636.409438] Lustre: Skipped 2 previous similar messages [ 8640.698265] Lustre: server umount lustre-MDT0000 complete [ 8641.506341] LustreError: 273161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.517094] LustreError: 273161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8644.607473] LustreError: 275012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632648 with bad export cookie 12753806125962888554 [ 8644.615678] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8644.620765] LustreError: 275012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8644.941516] Lustre: server umount lustre-MDT0001 complete [ 8656.819075] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8659.768456] Key type lgssc unregistered [ 8660.057477] LNet: 277475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.065054] LNetError: 277475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.083233] LNet: Removed LNI 192.168.204.106@tcp [ 8661.012595] Key type .llcrypt unregistered [ 8661.014862] Key type ._llcrypt unregistered [ 8678.590224] Key type ._llcrypt registered [ 8678.598606] Key type .llcrypt registered [ 8678.750165] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8680.325739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8680.383634] alg: No test for adler32 (adler32-zlib) [ 8681.432404] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8681.734777] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8683.448261] Key type lgssc registered [ 8684.372754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8691.463903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8691.475048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.929907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.471266] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8702.445438] LustreError: 278711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.463783] LustreError: 278711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8706.023234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.344676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8710.497118] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8717.907298] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8723.426904] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8730.039922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.417221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.518598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8737.589595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8744.396244] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8757.220064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8757.231492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8757.251450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.244977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.255461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.295350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8761.966694] Lustre: server umount lustre-OST0000 complete [ 8767.969433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8767.979044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.998411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8768.007715] Lustre: Skipped 1 previous similar message [ 8772.209750] Lustre: server umount lustre-MDT0000 complete [ 8773.604112] LustreError: 278717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.630539] LustreError: 278717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8776.181851] LustreError: 279471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632780 with bad export cookie 9564757322923047132 [ 8776.197583] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.202949] LustreError: 279471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8776.683772] Lustre: server umount lustre-MDT0001 complete [ 8788.229476] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8790.906861] Key type lgssc unregistered [ 8791.200733] LNet: 282413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8791.209386] LNetError: 282413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8791.227023] LNet: Removed LNI 192.168.204.106@tcp [ 8792.133477] Key type .llcrypt unregistered [ 8792.138312] Key type ._llcrypt unregistered [ 8803.155390] Key type ._llcrypt registered [ 8803.158302] Key type .llcrypt registered [ 8803.291619] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8819.332469] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8832.504910] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8833.227174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8833.243838] alg: No test for adler32 (adler32-zlib) [ 8834.299608] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8834.547809] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8836.192179] Key type lgssc registered [ 8837.199790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8869.232792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8869.265839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.613844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8870.649296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8870.797266] Lustre: lustre-MDT0000: new disk, initializing [ 8870.924244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.937689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8875.269890] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8884.657799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.815663] Lustre: 285966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.888902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.896474] Lustre: Skipped 1 previous similar message [ 8885.010206] Lustre: lustre-MDT0001: new disk, initializing [ 8885.122276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.158937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8885.168454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8889.862777] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8899.598841] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8905.342954] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.110045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.391427] Lustre: lustre-OST0000: new disk, initializing [ 8912.397139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8912.410396] Lustre: 287935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8912.540225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.134372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8919.082276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8919.090909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8919.188407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8926.023800] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8930.885700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8931.181051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8936.817659] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8937.106240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8939.490938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.540084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.624817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.627951] Lustre: Skipped 1 previous similar message [ 8945.560079] Lustre: server umount lustre-OST0000 complete [ 8949.732407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.747111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.758992] Lustre: Skipped 2 previous similar messages [ 8949.772458] Lustre: Skipped 1 previous similar message [ 8954.850780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.859588] Lustre: Skipped 1 previous similar message [ 8955.801609] Lustre: server umount lustre-MDT0000 complete [ 8958.810088] LustreError: 285959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787632962 with bad export cookie 6207368822081382838 [ 8958.814846] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.819400] LustreError: 285959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8959.047868] Lustre: server umount lustre-MDT0001 complete [ 8966.507524] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:42:48 (1787632968) [ 8973.091063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.619032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.655967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8978.918092] LustreError: 290347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.941662] LustreError: 290347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8984.036042] LustreError: 290346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8986.597482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.086527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.703390] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8997.475935] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9002.342830] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9008.468361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.834945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.556579] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9021.426990] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9032.674543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9032.690603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.718909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9034.723459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9034.746542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.984864] Lustre: server umount lustre-OST0000 complete [ 9043.425148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9043.434573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.458705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9043.463879] Lustre: Skipped 2 previous similar messages [ 9055.204190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9055.380268] Lustre: server umount lustre-MDT0000 complete [ 9059.218024] LustreError: 290329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633063 with bad export cookie 6207368822081383720 [ 9059.230241] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9059.588544] Lustre: server umount lustre-MDT0001 complete [ 9068.666283] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9071.499937] Key type lgssc unregistered [ 9071.735182] LNet: 293884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9071.751768] LNetError: 293884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9071.778887] LNet: Removed LNI 192.168.204.106@tcp [ 9072.456916] Key type .llcrypt unregistered [ 9072.460899] Key type ._llcrypt unregistered [ 9094.325501] Key type ._llcrypt registered [ 9094.328426] Key type .llcrypt registered [ 9094.418629] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9108.660542] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9109.502454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9109.519581] alg: No test for adler32 (adler32-zlib) [ 9110.586912] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 9110.875626] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9112.544160] Key type lgssc registered [ 9113.502424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9151.003600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9151.029385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.272311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.311150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9152.375521] Lustre: lustre-MDT0000: new disk, initializing [ 9152.434421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9152.459692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9156.652576] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9165.270202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.340863] Lustre: 297710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9165.395698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9165.400915] Lustre: Skipped 1 previous similar message [ 9165.492691] Lustre: lustre-MDT0001: new disk, initializing [ 9165.615293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9165.662392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9165.667559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9169.545252] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9180.494681] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9187.334848] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9194.241527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.587767] Lustre: lustre-OST2710: new disk, initializing [ 9194.591518] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9194.602510] Lustre: 299678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9194.735916] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9200.536745] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9201.717697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9201.732361] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9201.873983] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9208.884038] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9214.943646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9215.101850] Lustre: lustre-OST03e8: new disk, initializing [ 9215.105374] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9215.112848] Lustre: 300741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9215.184504] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9220.513539] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9224.817702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9224.833706] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9224.922137] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9227.380354] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9234.259511] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9234.444759] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9238.038377] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9238.209667] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9241.867968] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9242.083513] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9246.599457] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9246.901431] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9252.833660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9252.852399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.872206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9253.408455] Lustre: server umount lustre-MDT0000 complete [ 9255.926730] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.945218] Lustre: Skipped 2 previous similar messages [ 9257.243847] LustreError: 297703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633261 with bad export cookie 15490850097473651115 [ 9257.250418] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9257.258439] LustreError: 297703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9257.737784] Lustre: server umount lustre-MDT0001 complete [ 9272.830032] Lustre: server umount lustre-OST2710 complete [ 9276.393663] Lustre: 295509:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787633264/real 1787633264] req@00000000b6fc150a x1874469180210432/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787633280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9276.418290] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.616378] Lustre: server umount lustre-OST03e8 complete [ 9294.273668] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9304.071592] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9338.533467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.723776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9338.772490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9338.905091] Lustre: lustre-MDT0000: new disk, initializing [ 9339.001988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9339.034995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9343.364261] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9353.185815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.273900] Lustre: 305990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9353.347974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9353.352408] Lustre: Skipped 1 previous similar message [ 9353.440788] Lustre: lustre-MDT0001: new disk, initializing [ 9353.554694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9353.593569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9353.601783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9357.566425] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9366.901192] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9372.713385] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9380.422075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.746604] Lustre: lustre-OST0000: new disk, initializing [ 9380.751306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9380.757504] Lustre: 307957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9380.850964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9387.943790] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9388.159619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9388.176670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9388.418757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9398.805853] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9403.258258] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9403.466906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9407.700919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9407.897859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9414.114117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.126127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.126781] LustreError: Skipped 1 previous similar message [ 9414.128094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9414.156442] Lustre: Skipped 1 previous similar message [ 9415.524399] Lustre: server umount lustre-OST0000 complete [ 9419.237424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.239495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.257179] Lustre: Skipped 1 previous similar message [ 9419.269930] Lustre: Skipped 2 previous similar messages [ 9424.359859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.366819] Lustre: Skipped 1 previous similar message [ 9424.806871] Lustre: server umount lustre-MDT0000 complete [ 9427.338852] LustreError: 306783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633431 with bad export cookie 15490850097473652795 [ 9427.343968] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.345833] LustreError: 306783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9427.724655] Lustre: server umount lustre-MDT0001 complete [ 9436.777486] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:50:38 (1787633438) [ 9438.354915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9440.021957] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:50:41 (1787633441) [ 9443.589252] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9470.480184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.674359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9470.710492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9470.898272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.134784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9483.313923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.364604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9483.378417] Lustre: 311814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9487.173581] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9507.766245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.970168] Lustre: Denying initial registration attempt from nid 192.168.204.106@tcp, specified as failover [ 9507.975969] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9507.981840] LustreError: 313269:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9507.986416] LustreError: 313269:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9507.990378] LustreError: 313269:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9508.047160] Lustre: server umount lustre-OST0000 complete [ 9508.048821] LustreError: 313269:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9514.472213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9514.477227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9514.479748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9514.524339] Lustre: Skipped 1 previous similar message [ 9524.707737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9524.724406] Lustre: Skipped 4 previous similar messages [ 9528.288161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9528.439234] Lustre: server umount lustre-MDT0000 complete [ 9529.832190] LustreError: 313270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.855410] LustreError: 313270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9532.416375] LustreError: 312568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633536 with bad export cookie 15490850097473653677 [ 9532.425556] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9532.431804] LustreError: 312568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9533.025311] Lustre: server umount lustre-MDT0001 complete [ 9543.503391] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9546.380632] Key type lgssc unregistered [ 9546.722659] LNet: 314394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9546.729591] LNetError: 314394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9546.748325] LNet: Removed LNI 192.168.204.106@tcp [ 9547.532173] Key type .llcrypt unregistered [ 9547.533922] Key type ._llcrypt unregistered [ 9557.736539] Key type ._llcrypt registered [ 9557.742987] Key type .llcrypt registered [ 9557.870725] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9558.932485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9558.978771] alg: No test for adler32 (adler32-zlib) [ 9560.084348] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 9560.457380] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9562.120174] Key type lgssc registered [ 9563.243241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9610.880324] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9620.851871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9620.871651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.091209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9622.131853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9622.288384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9627.270676] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9636.303589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.367891] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9636.401610] Lustre: 317160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9636.587946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9641.460512] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9648.579731] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9654.034768] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.724314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9673.926344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9674.122392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.544476] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9687.201717] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9690.089298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9690.103964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9690.108903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9690.120472] Lustre: Skipped 1 previous similar message [ 9690.134889] Lustre: Skipped 1 previous similar message [ 9695.214048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9695.221460] Lustre: Skipped 1 previous similar message [ 9695.434448] Lustre: server umount lustre-OST0000 complete [ 9700.325122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.335071] Lustre: Skipped 1 previous similar message [ 9700.335244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9700.347510] Lustre: Skipped 1 previous similar message [ 9705.282133] Lustre: server umount lustre-MDT0000 complete [ 9708.741587] LustreError: 317901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633712 with bad export cookie 13278951884043574510 [ 9708.752389] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9708.758512] LustreError: 317901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9709.166949] Lustre: server umount lustre-MDT0001 complete [ 9720.143694] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9723.475778] Key type lgssc unregistered [ 9723.808630] LNet: 320735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9723.820250] LNetError: 320735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9723.846385] LNet: Removed LNI 192.168.204.106@tcp [ 9724.921327] Key type .llcrypt unregistered [ 9724.923313] Key type ._llcrypt unregistered [ 9751.364390] Key type ._llcrypt registered [ 9751.367899] Key type .llcrypt registered [ 9751.494454] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9752.831197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9752.940660] alg: No test for adler32 (adler32-zlib) [ 9754.070127] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 9754.446939] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9756.176189] Key type lgssc registered [ 9757.514709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9766.125689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9766.161968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9767.681318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9772.828752] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9777.125067] LustreError: 322156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.143789] LustreError: 322156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9782.029776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.244902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9782.455614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9787.861311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9796.454534] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9803.181043] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9812.016957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9812.433228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.910431] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9826.167679] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.051180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.058045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.071070] Lustre: Skipped 2 previous similar messages [ 9838.094099] Lustre: Skipped 2 previous similar messages [ 9841.782784] Lustre: server umount lustre-MDT0000 complete [ 9843.170969] LustreError: 322156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9843.180391] LustreError: 322156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9846.672761] LustreError: 322136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633850 with bad export cookie 17897688295612042438 [ 9846.693130] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.290176] LustreError: 322150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.290739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9848.316764] LustreError: 322150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9848.345374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9848.369825] Lustre: Skipped 1 previous similar message [ 9851.298028] LustreError: 322151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.307808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9851.334357] LustreError: 322151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9851.345864] Lustre: Skipped 1 previous similar message [ 9853.109961] Lustre: server umount lustre-MDT0001 complete [ 9862.125934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9871.882079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.659546] LustreError: 323988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.740935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.789573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9896.947436] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9902.055268] LustreError: 323989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.072463] LustreError: 323989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9906.169514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9907.816352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.479307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9913.868073] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9914.755837] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9923.369517] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9925.163764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9932.020910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9933.987493] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9943.009140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9943.019356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.032037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9944.557964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9946.547895] Lustre: server umount lustre-OST0000 complete [ 9954.274932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9954.288871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.323269] Lustre: Skipped 1 previous similar message [ 9954.336292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9954.353235] Lustre: Skipped 2 previous similar messages [ 9965.027128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9965.332902] Lustre: server umount lustre-MDT0000 complete [ 9970.148197] LustreError: 325754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.191431] LustreError: 325754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9971.489991] LustreError: 322135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787633975 with bad export cookie 17897688295612049585 [ 9971.501179] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9971.528777] LustreError: 322135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9971.972251] Lustre: server umount lustre-MDT0001 complete [ 9988.276761] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9991.065619] Key type lgssc unregistered [ 9991.423805] LNet: 328445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9991.439853] LNetError: 328445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9991.475216] LNet: Removed LNI 192.168.204.106@tcp [ 9992.594902] Key type .llcrypt unregistered [ 9992.597399] Key type ._llcrypt unregistered [10017.720366] Key type ._llcrypt registered [10017.724588] Key type .llcrypt registered [10017.834898] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10018.926059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10019.020497] alg: No test for adler32 (adler32-zlib) [10020.175223] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [10020.427693] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10022.168220] Key type lgssc registered [10023.422969] Lustre: Echo OBD driver; http://www.lustre.org/ [10031.945680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10031.978445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.542591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.257337] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10043.873656] LustreError: 329863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.902937] LustreError: 329863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10047.518164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.051183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10052.170668] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10059.068587] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10064.439234] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10067.388180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.399310] Lustre: Skipped 1 previous similar message [10067.406474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.410514] Lustre: Skipped 1 previous similar message [10072.546755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.553456] Lustre: Skipped 1 previous similar message [10073.712725] Lustre: server umount lustre-MDT0000 complete [10077.200894] LustreError: 329842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634081 with bad export cookie 6518247670101859345 [10077.213073] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.217145] LustreError: 329842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10077.597602] Lustre: server umount lustre-MDT0001 complete [10090.456899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10090.692898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10090.716824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10090.855561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.321724] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10103.425492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.559684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10103.611295] Lustre: 332462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10103.976733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10109.664570] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10117.451605] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10124.188465] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10134.083743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10134.383195] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10134.399766] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10134.421952] LustreError: 334346:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10134.428606] LustreError: 334346:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10134.434386] LustreError: 334346:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10134.533633] Lustre: server umount lustre-OST0000 complete [10134.538897] LustreError: 334346:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10145.385180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10145.603946] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10145.869982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10153.452400] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10154.149668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10161.491878] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10169.630629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.750262] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10169.896955] Lustre: lustre-OST0001: new disk, initializing [10169.904397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10169.921342] Lustre: 335887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10170.086777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10176.669102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10176.682766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10176.834210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10177.000484] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10184.420291] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10187.240019] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.240043] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10187.259932] Lustre: Skipped 1 previous similar message [10187.273312] LustreError: Skipped 1 previous similar message [10187.273889] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10192.367828] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10192.377105] Lustre: Skipped 2 previous similar messages [10193.448777] Lustre: server umount lustre-OST0001 complete [10197.476071] LustreError: 334811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.476297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.477953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10197.494755] LustreError: 334811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10197.533110] Lustre: Skipped 1 previous similar message [10202.596151] LustreError: 334814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.630250] LustreError: 334814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10203.287410] Lustre: server umount lustre-OST0000 complete [10207.719545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10207.732768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.739237] Lustre: Skipped 1 previous similar message [10207.753163] Lustre: Skipped 4 previous similar messages [10213.297882] Lustre: server umount lustre-MDT0000 complete [10216.835326] LustreError: 333200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634220 with bad export cookie 6518247670101859898 [10216.840316] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10216.844064] LustreError: 333200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10217.202957] Lustre: server umount lustre-MDT0001 complete [10228.800917] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10231.761960] Key type lgssc unregistered [10232.001617] LNet: 337849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10232.010463] LNetError: 337849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10233.071368] LNet: Removed LNI 192.168.204.106@tcp [10234.032157] Key type .llcrypt unregistered [10234.034130] Key type ._llcrypt unregistered [10304.956853] Key type ._llcrypt registered [10304.962381] Key type .llcrypt registered [10305.116499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [10323.455324] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10324.935773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10324.979831] alg: No test for adler32 (adler32-zlib) [10326.016723] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [10326.393221] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10328.104185] Key type lgssc registered [10330.401532] Lustre: Echo OBD driver; http://www.lustre.org/ [10367.873614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10367.915345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.142316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10369.180247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10369.269271] Lustre: lustre-MDT0000: new disk, initializing [10369.377357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10369.397710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10373.990512] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10385.129787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.312646] Lustre: 342791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10385.380418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10385.384182] Lustre: Skipped 1 previous similar message [10385.475955] Lustre: lustre-MDT0001: new disk, initializing [10385.596500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10385.639433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10385.648895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10390.073723] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10399.526416] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10405.480737] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10412.435107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10412.653583] Lustre: lustre-OST0000: new disk, initializing [10412.659090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10412.665635] Lustre: 344757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10412.782425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.777309] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10419.838099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10419.845948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10420.016164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10429.809208] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10434.341023] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10434.516306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10438.995725] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10439.276525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10445.795877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10445.799307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.806809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10445.817115] Lustre: Skipped 1 previous similar message [10445.829047] Lustre: Skipped 1 previous similar message [10447.480067] Lustre: server umount lustre-OST0000 complete [10456.052266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.059591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10456.087204] Lustre: Skipped 1 previous similar message [10459.128765] Lustre: server umount lustre-MDT0000 complete [10463.931741] LustreError: 342783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634467 with bad export cookie 465296467762858498 [10463.943464] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10463.950343] LustreError: 342783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10464.267964] Lustre: server umount lustre-MDT0001 complete [10474.149957] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:07:55 (1787634475) [10483.231346] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:08:04 (1787634484) [10491.263623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.641394] LustreError: 347513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.657152] LustreError: 347513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10491.722426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10495.666599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10496.997327] LustreError: 347514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.115064] LustreError: 347513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10503.537623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.053507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10509.484648] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10517.579687] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10523.814066] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.341460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10530.725154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10537.680784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10544.991727] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10555.490954] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10557.160499] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10558.607413] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10560.017156] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10561.550241] Lustre: Failing over lustre-MDT0000 [10561.739710] Lustre: server umount lustre-MDT0000 complete [10562.154104] LustreError: 350005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.557235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.593755] Lustre: Skipped 2 previous similar messages [10567.241286] LustreError: 347510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.265948] LustreError: 347510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10577.495864] LustreError: 347508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.510989] LustreError: 347508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10579.940609] Lustre: 341015:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787634567/real 1787634567] req@0000000002fc9d97 x1874470455285504/t0(0) o400->MGC192.168.204.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787634583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10579.964405] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10580.033111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10589.159030] LustreError: 341011:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f31797c6 x1874470455290624/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10589.512787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10589.565065] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10593.866108] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10593.982086] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10594.813879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10594.860596] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10594.919546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10602.383820] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10603.720146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10606.954658] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10608.616358] Lustre: Failing over lustre-MDT0000 [10608.911195] Lustre: server umount lustre-MDT0000 complete [10609.269377] LustreError: 350005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.295039] LustreError: 350005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10610.149537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.152298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10610.176661] LustreError: Skipped 1 previous similar message [10626.240708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.336678] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10626.576619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10626.614935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10630.376698] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10630.736779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10631.661094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10631.672028] Lustre: Skipped 2 previous similar messages [10631.686704] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10631.735123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10638.954596] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10640.373744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10647.010484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10647.013503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.030338] Lustre: Skipped 2 previous similar messages [10647.042268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10647.058681] Lustre: Skipped 3 previous similar messages [10652.149950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10652.155954] Lustre: Skipped 2 previous similar messages [10652.408354] Lustre: server umount lustre-MDT0000 complete [10655.557460] LustreError: 347493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634659 with bad export cookie 465296467762861207 [10655.558549] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10655.580417] LustreError: 347493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10655.812693] Lustre: server umount lustre-MDT0001 complete [10669.789929] Lustre: server umount lustre-OST0000 complete [10685.468663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.009838] LustreError: 353851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.019616] LustreError: 353851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10691.936797] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10701.670105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10701.953392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10701.960662] Lustre: Skipped 1 previous similar message [10705.705722] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10714.090905] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10720.090385] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10727.595602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.995784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10734.152455] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10740.692713] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10751.462580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10751.472864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.484865] LustreError: Skipped 1 previous similar message [10751.512444] Lustre: Skipped 2 previous similar messages [10751.522762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10751.532184] Lustre: Skipped 1 previous similar message [10752.170221] Lustre: server umount lustre-OST0000 complete [10761.697663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.705777] Lustre: Skipped 3 previous similar messages [10769.888251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10770.053957] Lustre: server umount lustre-MDT0000 complete [10771.948561] LustreError: 353851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.964632] LustreError: 353851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10773.833837] LustreError: 355695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634777 with bad export cookie 465296467762861956 [10773.836326] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.850134] LustreError: 355695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10774.303909] Lustre: server umount lustre-MDT0001 complete [10785.768541] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10789.401848] Key type lgssc unregistered [10789.740079] LNet: 357387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10789.757329] LNetError: 357387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10789.775966] LNet: Removed LNI 192.168.204.106@tcp [10790.746953] Key type .llcrypt unregistered [10790.749740] Key type ._llcrypt unregistered [10815.222549] Key type ._llcrypt registered [10815.228394] Key type .llcrypt registered [10815.337537] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10816.216875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10816.325940] alg: No test for adler32 (adler32-zlib) [10817.371169] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [10817.697615] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10819.416097] Key type lgssc registered [10820.759635] Lustre: Echo OBD driver; http://www.lustre.org/ [10829.078793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10829.104942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.668814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10835.636273] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10841.065431] LustreError: 358904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.092841] LustreError: 358904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10844.983439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10845.461895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10850.068533] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10857.316933] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10862.768717] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10869.681968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10870.018962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10875.130453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10875.640509] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10881.934108] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10890.723150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.726077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.741235] Lustre: Skipped 2 previous similar messages [10890.751286] Lustre: Skipped 2 previous similar messages [10895.843212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10895.847293] Lustre: Skipped 2 previous similar messages [10896.631283] Lustre: server umount lustre-MDT0000 complete [10899.889524] LustreError: 358885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787634903 with bad export cookie 17283287187496508569 [10899.898848] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.902567] LustreError: 358885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10900.174910] Lustre: server umount lustre-MDT0001 complete [10913.331156] Lustre: server umount lustre-OST0000 complete [10927.417685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.076487] LustreError: 362612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.174770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10933.023545] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10933.229606] LustreError: 362613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.348420] LustreError: 362612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.477232] LustreError: 362613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.101907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.503482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10949.127468] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10956.018805] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10960.873978] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10967.022428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10967.335495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10972.614720] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10973.497111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10979.165296] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10994.148293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.154945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10994.168211] Lustre: Skipped 2 previous similar messages [10994.180737] Lustre: Skipped 1 previous similar message [10997.105324] Lustre: server umount lustre-MDT0000 complete [10999.267281] LustreError: 363341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.292215] LustreError: 363341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11000.466715] LustreError: 362592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635004 with bad export cookie 17283287187496509689 [11000.474648] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.476693] LustreError: 362592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11000.757894] Lustre: server umount lustre-MDT0001 complete [11014.862361] Lustre: server umount lustre-OST0000 complete [11029.265299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.625728] LustreError: 366523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.698243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11033.185352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11040.018264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.703586] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11049.285210] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11054.498202] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11061.514613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11061.948741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11061.960046] Lustre: Skipped 1 previous similar message [11068.078342] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11074.240994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11075.053806] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11079.661359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11079.662132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11079.662786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11079.662792] Lustre: Skipped 1 previous similar message [11079.672892] Lustre: Skipped 1 previous similar message [11081.508928] Lustre: server umount lustre-OST0000 complete [11086.310773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11086.318814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.328503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11086.331747] Lustre: Skipped 1 previous similar message [11091.368816] Lustre: server umount lustre-MDT0000 complete [11094.794120] LustreError: 366505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635098 with bad export cookie 17283287187496510711 [11094.807951] LustreError: 366505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11094.810883] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.104408] Lustre: server umount lustre-MDT0001 complete [11105.899251] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11108.359849] Key type lgssc unregistered [11108.622380] LNet: 370056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11108.635812] LNetError: 370056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11108.653903] LNet: Removed LNI 192.168.204.106@tcp [11109.350642] Key type .llcrypt unregistered [11109.353756] Key type ._llcrypt unregistered [11132.053159] Key type ._llcrypt registered [11132.055706] Key type .llcrypt registered [11132.143940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11133.066779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11133.157919] alg: No test for adler32 (adler32-zlib) [11134.376466] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [11134.697850] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11136.392305] Key type lgssc registered [11137.677257] Lustre: Echo OBD driver; http://www.lustre.org/ [11145.821636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11145.844789] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.860887] LustreError: 371500:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11145.892480] LustreError: 371500:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11145.916172] LustreError: 371500:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11145.930550] LustreError: 371500:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11145.952949] LustreError: 371500:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11153.998678] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11154.004648] LustreError: 371897:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11154.022729] LustreError: 371897:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11154.032402] LustreError: 371897:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11154.037518] LustreError: 371897:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11154.052043] LustreError: 371897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11173.088456] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11176.174392] Key type lgssc unregistered [11176.448613] LNet: 372903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11176.453644] LNetError: 372903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11176.471385] LNet: Removed LNI 192.168.204.106@tcp [11177.317036] Key type .llcrypt unregistered [11177.321585] Key type ._llcrypt unregistered [11194.248241] Key type ._llcrypt registered [11194.250388] Key type .llcrypt registered [11194.364415] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [11206.794867] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11207.933170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11207.980760] alg: No test for adler32 (adler32-zlib) [11209.031385] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [11209.203958] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11210.881995] Key type lgssc registered [11212.011065] Lustre: Echo OBD driver; http://www.lustre.org/ [11243.841953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11243.886816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.054533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11245.086810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11245.156622] Lustre: lustre-MDT0000: new disk, initializing [11245.206140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11245.224443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11249.289965] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11259.098446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.197845] Lustre: 376040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11259.250690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11259.255313] Lustre: Skipped 1 previous similar message [11259.337974] Lustre: lustre-MDT0001: new disk, initializing [11259.410622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11259.451036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11259.465186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11263.487355] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11272.402783] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11277.347285] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11283.117515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.419184] Lustre: lustre-OST0000: new disk, initializing [11283.430347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11283.440791] Lustre: 378008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11283.523889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11288.642138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11288.657894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11288.821654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11289.120402] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11299.644990] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11305.398950] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11305.792037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11310.736940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11310.999188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11314.657258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11314.669725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11314.676833] LustreError: Skipped 1 previous similar message [11314.677847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11314.706894] Lustre: Skipped 1 previous similar message [11318.931326] Lustre: server umount lustre-OST0000 complete [11324.900407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11324.906294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11324.911631] Lustre: Skipped 1 previous similar message [11324.914919] Lustre: Skipped 1 previous similar message [11328.655398] Lustre: server umount lustre-MDT0000 complete [11331.955095] LustreError: 378007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635335 with bad export cookie 16573823926266214475 [11331.955904] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11331.969851] LustreError: 378007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11332.277792] Lustre: server umount lustre-MDT0001 complete [11339.151735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11340.450930] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:22:22 (1787635342) [11345.877283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.209926] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.224168] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11346.281238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.785369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11351.521836] LustreError: 380468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.647453] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.354404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.727478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11362.302500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11369.342343] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11374.126891] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11379.725772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.029549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11385.326366] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11391.454685] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11398.597192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11398.744681] Lustre: lustre-OST0001: new disk, initializing [11398.747832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11398.753442] Lustre: 383300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11398.837341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11405.193849] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11406.978259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11406.989413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11407.087847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11415.015273] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11421.797879] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [11421.808199] Lustre: Skipped 1 previous similar message [11422.693868] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11422.703694] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.719593] LustreError: Skipped 1 previous similar message [11424.999398] Lustre: server umount lustre-OST0001 complete [11426.898289] LustreError: 382291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.947968] LustreError: 382291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.970211] LustreError: 382291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11443.178286] LustreError: 382289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.200044] LustreError: 382289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11444.707053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11444.717395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.729217] Lustre: Skipped 1 previous similar message [11444.738742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11444.748860] Lustre: Skipped 2 previous similar messages [11447.778042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11449.914713] Lustre: server umount lustre-OST0000 complete [11454.946632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11454.964779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.980835] Lustre: Skipped 1 previous similar message [11454.983841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11454.986707] Lustre: Skipped 3 previous similar messages [11463.650923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11463.661366] Lustre: Skipped 3 previous similar messages [11468.256582] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11468.601328] Lustre: server umount lustre-MDT0000 complete [11468.777178] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.814775] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11472.969993] LustreError: 380449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635476 with bad export cookie 16573823926266215350 [11472.971362] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.999230] LustreError: 380449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11479.404699] Lustre: server umount lustre-MDT0001 complete [11491.316834] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11494.875796] Key type lgssc unregistered [11495.252631] LNet: 385294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11495.263895] LNetError: 385294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11495.289302] LNet: Removed LNI 192.168.204.106@tcp [11496.273207] Key type .llcrypt unregistered [11496.275339] Key type ._llcrypt unregistered [11548.929994] Key type ._llcrypt registered [11548.932893] Key type .llcrypt registered [11549.095435] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11550.593499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11550.634893] alg: No test for adler32 (adler32-zlib) [11551.764161] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [11552.072141] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11553.752176] Key type lgssc registered [11554.719157] Lustre: Echo OBD driver; http://www.lustre.org/ [11561.685734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11561.706498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11562.992433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11563.022419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11563.193652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11567.604535] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11575.690452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.754552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11575.776278] Lustre: 388091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11576.089985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11581.034279] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11589.134668] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11594.315208] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11596.771326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11596.782245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11596.795903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11599.331815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11599.333384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.206289] Lustre: server umount lustre-MDT0000 complete [11606.140169] LustreError: 388084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635610 with bad export cookie 14240991029512513557 [11606.151609] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11606.152167] LustreError: 388084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11606.528317] Lustre: server umount lustre-MDT0001 complete [11616.219767] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:26:57 (1787635617) [11623.823033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.375385] LustreError: 390601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.401985] LustreError: 390601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11624.441086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11629.349303] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11629.549107] LustreError: 390602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11634.659108] LustreError: 390602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.895918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.282195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11644.101482] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11653.083351] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11659.629540] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11666.843258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11667.024848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11667.190307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11673.188179] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11679.842592] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.622993] Lustre: Permanently deactivating lustre-OST0000 [11686.642272] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11692.598651] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11693.707973] LustreError: 393540:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11693.718917] LustreError: 393540:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11693.726983] LustreError: 393540:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11696.582956] LustreError: 393589:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11696.595848] LustreError: 393589:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11704.806436] Lustre: server umount lustre-OST0000 complete [11707.600819] LustreError: 393838:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11707.609147] LustreError: 393838:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11709.411447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.415989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.423209] Lustre: Skipped 1 previous similar message [11709.430338] Lustre: Skipped 2 previous similar messages [11714.530865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.541145] Lustre: Skipped 1 previous similar message [11719.654488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11719.661070] Lustre: Skipped 1 previous similar message [11719.756425] Lustre: server umount lustre-MDT0000 complete [11723.567291] LustreError: 390583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635727 with bad export cookie 14240991029512514138 [11723.574108] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11723.584173] LustreError: 390583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11734.184739] Lustre: server umount lustre-MDT0001 complete [11743.188332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11748.906951] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11751.972567] LustreError: 394997:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11759.083434] Lustre: 395336:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.106@tcp [11761.243521] Lustre: server umount MGS complete [11773.030202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.495321] LustreError: 396062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.512769] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11773.528152] Lustre: Skipped 1 previous similar message [11773.625636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11778.750386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11779.045642] LustreError: 396063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.868543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.117973] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11793.426554] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11801.222316] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11806.106835] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11808.054201] Lustre: Permanently reactivating lustre-OST0000 [11808.069735] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11808.075263] Lustre: Skipped 2 previous similar messages [11838.432177] LustreError: 397893:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11846.107979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11846.501934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11846.506480] Lustre: Skipped 1 previous similar message [11853.112597] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11861.264296] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11868.129661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11868.137368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11868.161076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11869.155413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11881.952127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11882.235445] Lustre: server umount lustre-OST0000 complete [11889.638256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.662218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11889.663337] Lustre: Skipped 2 previous similar messages [11889.685742] Lustre: Skipped 7 previous similar messages [11893.012476] Lustre: server umount lustre-MDT0000 complete [11894.755266] LustreError: 396790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.790485] LustreError: 396790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11896.732417] LustreError: 396043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787635900 with bad export cookie 14240991029512515188 [11896.734859] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11896.750365] LustreError: 396043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11896.984214] Lustre: server umount lustre-MDT0001 complete [11909.275096] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11912.985200] Key type lgssc unregistered [11913.288507] LNet: 400482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11913.304129] LNetError: 400482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11913.315177] LNet: Removed LNI 192.168.204.106@tcp [11914.439093] Key type .llcrypt unregistered [11914.444663] Key type ._llcrypt unregistered [11935.958469] Key type ._llcrypt registered [11935.965575] Key type .llcrypt registered [11936.183601] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [11956.821770] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11957.974572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11958.024517] alg: No test for adler32 (adler32-zlib) [11959.094375] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [11959.250101] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11960.952163] Key type lgssc registered [11962.091862] Lustre: Echo OBD driver; http://www.lustre.org/ [12000.867364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12000.894941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.244860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12002.285475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12002.474374] Lustre: lustre-MDT0000: new disk, initializing [12002.581598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12002.606588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12008.159178] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12023.109948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12023.409133] Lustre: 403644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12023.599623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12023.605327] Lustre: Skipped 1 previous similar message [12023.839688] Lustre: lustre-MDT0001: new disk, initializing [12024.108601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12024.233916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12024.264761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12032.475455] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12042.711233] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12049.235110] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12057.197369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12057.612804] Lustre: lustre-OST0000: new disk, initializing [12057.626463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12057.640377] Lustre: 405610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12057.752931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12065.348218] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12066.409608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12066.421178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12066.566487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12075.552612] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12080.741629] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12080.970971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12085.550468] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12085.779606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12092.386831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12092.395062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12092.397281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12092.428400] Lustre: Skipped 1 previous similar message [12094.204284] Lustre: server umount lustre-OST0000 complete [12101.095626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12101.107033] LustreError: Skipped 1 previous similar message [12101.111773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.141415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12101.143735] Lustre: Skipped 1 previous similar message [12102.626650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.627819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.660487] Lustre: Skipped 1 previous similar message [12104.810134] Lustre: server umount lustre-MDT0000 complete [12108.872195] LustreError: 404446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787636112 with bad export cookie 8660337334679203932 [12108.878155] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12108.895146] LustreError: 404446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12109.310949] Lustre: server umount lustre-MDT0001 complete [12119.021633] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12120.760512] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:35:22 (1787636122) [12128.089141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.578247] LustreError: 408072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.595585] LustreError: 408072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12128.707391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12133.459985] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12133.863303] LustreError: 408073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12138.981094] LustreError: 408072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.901874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12143.339224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.378232] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12156.805167] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12163.050839] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12170.813562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12171.162381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12177.270984] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12185.873042] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12188.046788] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12191.780399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12198.887970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.906706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12198.907118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12198.913854] LustreError: Skipped 1 previous similar message [12198.927790] Lustre: Skipped 1 previous similar message [12203.418665] Lustre: server umount lustre-OST0000 complete [12209.122848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12209.131924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12209.143927] Lustre: Skipped 2 previous similar messages [12209.178119] Lustre: Skipped 1 previous similar message [12212.828635] Lustre: server umount lustre-MDT0000 complete [12214.249566] LustreError: 408072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.265247] LustreError: 408072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12216.426723] LustreError: 408053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787636220 with bad export cookie 8660337334679204814 [12216.430881] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12216.448680] LustreError: 408053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12216.903708] Lustre: server umount lustre-MDT0001 complete [12229.977612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12233.674366] Key type lgssc unregistered [12234.117538] LNet: 411801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12234.124079] LNetError: 411801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12234.151712] LNet: Removed LNI 192.168.204.106@tcp [12235.456150] Key type .llcrypt unregistered [12235.458929] Key type ._llcrypt unregistered [12262.885746] Key type ._llcrypt registered [12262.886910] Key type .llcrypt registered [12263.110790] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12263.628976] Key type .llcrypt unregistered [12263.630710] Key type ._llcrypt unregistered [12283.300244] Key type ._llcrypt registered [12283.303275] Key type .llcrypt registered [12283.434311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [12284.905500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12285.178473] alg: No test for adler32 (adler32-zlib) [12286.356949] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [12286.646760] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [12288.312920] Key type lgssc registered [12289.688369] Lustre: Echo OBD driver; http://www.lustre.org/ [12297.178671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12297.195331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.791282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12303.250883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12308.966385] LustreError: 414019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.978759] LustreError: 414019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12312.868949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12313.158293] LustreError: 414020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12313.182655] LustreError: 414020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12313.185711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12318.178492] LustreError: 414016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.205473] LustreError: 414016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12321.743595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12323.302134] LustreError: 414020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.314351] LustreError: 414020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12323.335816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12328.420376] LustreError: 414015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.444363] LustreError: 414015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12330.996368] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12338.390809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.735633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12343.389574] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12354.066319] Lustre: server umount lustre-OST0000 complete [12358.675886] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12358.688963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.717555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.258134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.263782] Lustre: Skipped 3 previous similar messages [12368.432442] Lustre: server umount lustre-MDT0000 complete [12369.386803] LustreError: 414019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12369.404673] LustreError: 414019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12370.408533] Lustre: 413330:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787636357/real 1787636357] req@000000004c0e130d x1874472510449152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787636373 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12370.441412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12370.458438] Lustre: Skipped 1 previous similar message [12372.167287] LustreError: 414000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787636376 with bad export cookie 13522953001358031739 [12372.172907] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12372.178157] LustreError: 414000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12372.539487] Lustre: server umount lustre-MDT0001 complete [12383.511452] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12386.737907] Key type lgssc unregistered [12387.053613] LNet: 417170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12387.061348] LNetError: 417170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12387.087382] LNet: Removed LNI 192.168.204.106@tcp [12387.957828] Key type .llcrypt unregistered [12387.962655] Key type ._llcrypt unregistered [12417.041076] Key type ._llcrypt registered [12417.046947] Key type .llcrypt registered [12417.126132] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [12418.888365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.941898] alg: No test for adler32 (adler32-zlib) [12420.040611] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [12420.366719] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [12422.049808] Key type lgssc registered [12423.608262] Lustre: Echo OBD driver; http://www.lustre.org/ [12433.092257] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12433.122056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12448.801413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12453.920638] LustreError: 418567:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [12480.608230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12488.118554] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12496.828611] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12504.512173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12505.118505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12505.215866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12510.152514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12510.200646] LustreError: 418593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12510.221390] LustreError: 418593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12515.300019] LustreError: 419604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.318980] LustreError: 419604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12519.401315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12519.969931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12524.633798] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12533.198716] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12540.273293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12548.312983] Lustre: server umount lustre-OST0000 complete [12554.733217] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12554.755177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12554.798266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12555.746459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12555.750207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12555.785042] Lustre: Skipped 1 previous similar message [12560.876351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12560.894586] Lustre: Skipped 1 previous similar message [12563.009316] Lustre: server umount lustre-MDT0000 complete [12565.989528] LustreError: 419610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12566.006913] LustreError: 419610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12566.868578] LustreError: 418573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787636570 with bad export cookie 15866786274455637482 [12566.899618] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12576.413745] Lustre: server umount lustre-MDT0001 complete [12588.228037] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12591.112353] Key type lgssc unregistered [12591.352266] LNet: 422141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12591.362990] LNetError: 422141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12591.389838] LNet: Removed LNI 192.168.204.106@tcp [12592.351184] Key type .llcrypt unregistered [12592.353402] Key type ._llcrypt unregistered [12618.181570] Key type ._llcrypt registered [12618.184845] Key type .llcrypt registered [12618.299515] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [12619.401359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12619.569830] alg: No test for adler32 (adler32-zlib) [12620.665874] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [12620.966119] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [12622.736836] Key type lgssc registered [12623.966766] Lustre: Echo OBD driver; http://www.lustre.org/ [12632.139868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12632.165271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12633.659565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12638.744535] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12643.822085] LustreError: 423574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.847917] LustreError: 423574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12646.890605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.368192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12652.931122] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12662.242024] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12669.223790] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12676.579799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12676.816747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12678.902400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12683.062548] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12691.476047] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12697.571519] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12697.583824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12697.592307] Lustre: Skipped 1 previous similar message [12697.602601] Lustre: Skipped 2 previous similar messages [12698.730605] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [12702.100928] Lustre: server umount lustre-MDT0000 complete [12702.696375] LustreError: 423571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12702.720824] LustreError: 423571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12703.820605] LustreError: 426017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.846020] LustreError: 423556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.6@tcp arrived at 1787636708 with bad export cookie 2275387373063624478 [12706.901287] LustreError: 423554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787636710 with bad export cookie 2275387373063623589 [12706.913474] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12706.928154] LustreError: 423554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12707.810459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12707.815370] LustreError: 423575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.815387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.842224] LustreError: 423575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12707.865075] Lustre: Skipped 1 previous similar message [12707.870261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.841790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.845116] LustreError: 424302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.882205] LustreError: 424302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12713.151346] Lustre: server umount lustre-OST0000 complete [12720.455446] Lustre: server umount lustre-MDT0001 complete [12733.504236] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12736.695749] Key type lgssc unregistered [12737.095841] LNet: 427265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12737.116917] LNetError: 427265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12737.144911] LNet: Removed LNI 192.168.204.106@tcp [12738.088532] Key type .llcrypt unregistered [12738.091207] Key type ._llcrypt unregistered [12764.339824] Key type ._llcrypt registered [12764.343834] Key type .llcrypt registered [12764.515308] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [12765.947128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12766.073182] alg: No test for adler32 (adler32-zlib) [12767.323469] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [12767.676517] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [12769.441049] Key type lgssc registered [12770.516929] Lustre: Echo OBD driver; http://www.lustre.org/ [12779.078618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12779.098765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12780.503070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12786.683745] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12790.763176] LustreError: 428686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.789126] LustreError: 428686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12795.877425] LustreError: 428687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12796.752199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.579233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12802.808339] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12810.916951] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12817.415463] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12824.763347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12825.235751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12826.305097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12831.818537] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12840.459577] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12847.688848] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12848.102463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12852.824963] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12852.835155] Lustre: Skipped 2 previous similar messages [12857.943631] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12857.955176] Lustre: Skipped 2 previous similar messages [12859.360126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12859.581103] Lustre: server umount lustre-MDT0001 complete [12863.462564] LustreError: 428683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12863.494169] LustreError: 428683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12867.557977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12867.572404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.588529] Lustre: Skipped 1 previous similar message [12867.606881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12867.619222] Lustre: Skipped 2 previous similar messages [12868.581461] LustreError: 428687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12873.020847] Lustre: server umount lustre-OST0000 complete [12873.709046] LustreError: 429414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.264787] Lustre: server umount lustre-MDT0000 complete [12892.226650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [12895.932049] Key type lgssc unregistered [12896.193781] LNet: 432373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12896.207807] LNetError: 432373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12896.231391] LNet: Removed LNI 192.168.204.106@tcp [12897.218890] Key type .llcrypt unregistered [12897.220625] Key type ._llcrypt unregistered [12925.239324] Key type ._llcrypt registered [12925.241582] Key type .llcrypt registered [12925.439893] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [12941.673741] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [12942.701758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12942.732546] alg: No test for adler32 (adler32-zlib) [12943.956935] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [12944.359231] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [12946.096215] Key type lgssc registered [12947.349626] Lustre: Echo OBD driver; http://www.lustre.org/ [12990.373037] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [13005.523297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13005.551438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.793164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13006.833965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13006.921528] Lustre: lustre-MDT0000: new disk, initializing [13007.071829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13007.110743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13012.183883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13028.831831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13029.559174] Lustre: lustre-OST0000: new disk, initializing [13029.567331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13029.580026] Lustre: Skipped 1 previous similar message [13029.590352] Lustre: 437407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13029.821039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13039.290109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13039.304101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13039.333850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13040.362926] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13052.223513] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13067.824022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.962430] Lustre: 436388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13068.083681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13068.171852] Lustre: lustre-MDT0001: new disk, initializing [13068.293046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13068.331951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13068.348254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13073.225425] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13081.903517] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13082.078696] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13085.748313] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13086.056711] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13094.888287] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13094.896166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13094.923119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13094.974871] Lustre: Skipped 1 previous similar message [13095.545615] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [13095.558054] Lustre: Skipped 1 previous similar message [13100.003576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13100.011833] Lustre: Skipped 1 previous similar message [13105.135311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13105.142905] Lustre: Skipped 2 previous similar messages [13106.146939] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13106.484637] Lustre: server umount lustre-MDT0001 complete [13115.369942] LustreError: 436395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13115.392847] LustreError: 436395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13120.495074] LustreError: 436400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.536457] LustreError: 436400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13125.603028] LustreError: 439038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13125.605387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13125.656595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13125.664206] Lustre: Skipped 1 previous similar message [13130.736523] LustreError: 437391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13130.762930] LustreError: 437391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13130.914676] Lustre: server umount lustre-OST0000 complete [13141.905327] Lustre: server umount lustre-MDT0000 complete [13158.839700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [13162.286580] Key type lgssc unregistered [13162.531577] LNet: 441173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13162.534899] LNetError: 441173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13162.559258] LNet: Removed LNI 192.168.204.106@tcp [13163.420229] Key type .llcrypt unregistered [13163.435297] Key type ._llcrypt unregistered [13205.480508] Key type ._llcrypt registered [13205.482686] Key type .llcrypt registered [13205.642608] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [13206.043194] Key type .llcrypt unregistered [13206.050917] Key type ._llcrypt unregistered [13223.366128] Key type ._llcrypt registered [13223.367979] Key type .llcrypt registered [13223.529578] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [13224.493596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13224.521635] alg: No test for adler32 (adler32-zlib) [13225.438587] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [13225.569715] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [13227.427218] Key type lgssc registered [13229.247888] Lustre: Echo OBD driver; http://www.lustre.org/ [13259.409155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13259.429638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13260.689361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13260.745372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13260.829916] Lustre: lustre-MDT0000: new disk, initializing [13260.909755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13260.924627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13265.905891] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13277.483856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13277.652358] Lustre: 444554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13277.772643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13277.783833] Lustre: Skipped 1 previous similar message [13277.850208] Lustre: lustre-MDT0001: new disk, initializing [13277.930299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13277.969181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13277.987272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13283.751987] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13293.876533] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13300.044900] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13307.803714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13308.079506] Lustre: lustre-OST0000: new disk, initializing [13308.083092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13308.094894] Lustre: 446523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13308.191835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13315.791959] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13317.722440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13317.746813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13317.908035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13326.152802] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13333.477798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13333.493948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13333.516635] Lustre: Skipped 2 previous similar messages [13333.541634] Lustre: Skipped 2 previous similar messages [13338.187549] Lustre: server umount lustre-MDT0000 complete [13343.172544] LustreError: 444547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787637347 with bad export cookie 18439547847540711232 [13343.183969] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13343.188108] LustreError: 444547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13343.627954] Lustre: server umount lustre-MDT0001 complete [13360.226827] Lustre: server umount lustre-OST0000 complete [13373.301488] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:56:14 (1787637374) [13380.149158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13380.532292] LustreError: 448552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13380.558037] LustreError: 448552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13380.648537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13384.467425] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13385.707178] LustreError: 448553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.821540] LustreError: 448552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13392.829953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13393.208725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13399.475642] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13409.177102] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13415.123172] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13434.542032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13434.761875] Lustre: Found index 0 for lustre-OST0000, updating log [13434.930650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13440.656637] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13447.976556] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13451.318287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13459.427512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13459.439199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13459.454499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13460.521631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13460.536185] Lustre: Skipped 2 previous similar messages [13464.358989] Lustre: server umount lustre-OST0000 complete [13481.956314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13481.969581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13481.974420] Lustre: Skipped 2 previous similar messages [13481.995779] Lustre: Skipped 1 previous similar message [13486.114235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13486.121462] Lustre: Skipped 1 previous similar message [13487.122785] Lustre: server umount lustre-MDT0000 complete [13487.155249] LustreError: 448547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13487.175165] LustreError: 448547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13491.306313] LustreError: 448534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787637495 with bad export cookie 18439547847540712625 [13491.313057] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13491.323728] LustreError: 448534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13491.647393] Lustre: server umount lustre-MDT0001 complete [13500.532763] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:58:22 (1787637502) [13501.094865] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [14125.367639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14125.925052] LustreError: 454979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14126.056195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14130.925236] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14131.170561] LustreError: 454980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14131.189973] LustreError: 454980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14135.266830] LustreError: 454979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14139.655898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14140.242260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14144.663399] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14151.808693] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14157.392148] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14177.207991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14177.554901] Lustre: Found index 0 for lustre-OST0000, updating log [14177.750753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14180.001285] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [14184.961856] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14192.811471] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14194.699202] Lustre: server umount lustre-OST0000 complete [14199.842163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14199.853079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14200.867327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14200.874621] Lustre: Skipped 2 previous similar messages [14202.914856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14202.930798] Lustre: Skipped 3 previous similar messages [14205.139543] Lustre: server umount lustre-MDT0000 complete [14206.058342] LustreError: 455710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14206.098058] LustreError: 455710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14209.373445] LustreError: 454961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787638213 with bad export cookie 18439547847540713703 [14209.379972] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14209.387405] LustreError: 454961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14209.803216] Lustre: server umount lustre-MDT0001 complete [14239.393387] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [14242.809387] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [14254.064806] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [14294.303309] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:11:35 (1787638295) [14311.359560] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [14314.439421] Key type lgssc unregistered [14314.683572] LNet: 461597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14314.692492] LNetError: 461597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14439.659667] LNet: Removed LNI 192.168.204.106@tcp [14440.668308] Key type .llcrypt unregistered [14440.675373] Key type ._llcrypt unregistered [14458.236172] Key type ._llcrypt registered [14458.241910] Key type .llcrypt registered [14458.411996] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [14460.050608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14460.079711] alg: No test for adler32 (adler32-zlib) [14461.199860] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [14461.585440] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [14463.280184] Key type lgssc registered [14464.516613] Lustre: Echo OBD driver; http://www.lustre.org/ [14469.741602] LNet: Added LNI 192.168.204.106@tcp100 [8/256/0/180] [14470.595631] LNet: Added LNI 192.168.204.106@tcp101 [8/256/0/180] [14471.678048] LNet: Added LNI 192.168.204.106@tcp102 [8/256/0/180] [14472.694773] LNet: Added LNI 192.168.204.106@tcp103 [8/256/0/180] [14473.766493] LNet: Added LNI 192.168.204.106@tcp104 [8/256/0/180] [14474.642351] LNet: Added LNI 192.168.204.106@tcp105 [8/256/0/180] [14475.633053] LNet: Added LNI 192.168.204.106@tcp106 [8/256/0/180] [14476.669831] LNet: Added LNI 192.168.204.106@tcp107 [8/256/0/180] [14477.742214] LNet: Added LNI 192.168.204.106@tcp108 [8/256/0/180] [14478.848699] LNet: Added LNI 192.168.204.106@tcp109 [8/256/0/180] [14480.020105] LNet: Added LNI 192.168.204.106@tcp110 [8/256/0/180] [14481.069078] LNet: Added LNI 192.168.204.106@tcp111 [8/256/0/180] [14482.027232] LNet: Added LNI 192.168.204.106@tcp112 [8/256/0/180] [14483.060684] LNet: Added LNI 192.168.204.106@tcp113 [8/256/0/180] [14484.175046] LNet: Added LNI 192.168.204.106@tcp114 [8/256/0/180] [14485.310874] LNet: Added LNI 192.168.204.106@tcp115 [8/256/0/180] [14486.268401] LNet: Added LNI 192.168.204.106@tcp116 [8/256/0/180] [14487.525393] LNet: Added LNI 192.168.204.106@tcp117 [8/256/0/180] [14488.577295] LNet: Added LNI 192.168.204.106@tcp118 [8/256/0/180] [14489.806780] LNet: Added LNI 192.168.204.106@tcp119 [8/256/0/180] [14490.902984] LNet: Added LNI 192.168.204.106@tcp120 [8/256/0/180] [14491.926991] LNet: Added LNI 192.168.204.106@tcp121 [8/256/0/180] [14492.946378] LNet: Added LNI 192.168.204.106@tcp122 [8/256/0/180] [14493.836387] LNet: Added LNI 192.168.204.106@tcp123 [8/256/0/180] [14494.787704] LNet: Added LNI 192.168.204.106@tcp124 [8/256/0/180] [14495.838702] LNet: Added LNI 192.168.204.106@tcp125 [8/256/0/180] [14496.898649] LNet: Added LNI 192.168.204.106@tcp126 [8/256/0/180] [14497.831237] LNet: Added LNI 192.168.204.106@tcp127 [8/256/0/180] [14498.871942] LNet: Added LNI 192.168.204.106@tcp128 [8/256/0/180] [14499.901816] LNet: Added LNI 192.168.204.106@tcp129 [8/256/0/180] [14500.931964] LNet: Added LNI 192.168.204.106@tcp130 [8/256/0/180] [14502.028899] LNet: Added LNI 192.168.204.106@tcp131 [8/256/0/180] [14503.117111] LNet: Added LNI 192.168.204.106@tcp132 [8/256/0/180] [14504.177905] LNet: Added LNI 192.168.204.106@tcp133 [8/256/0/180] [14505.080622] LNet: Added LNI 192.168.204.106@tcp134 [8/256/0/180] [14506.230926] LNet: Added LNI 192.168.204.106@tcp135 [8/256/0/180] [14512.674762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14512.701900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14513.847726] Lustre: 464634:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [14513.933113] LNetError: 462091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14513.961716] LNetError: Protocol error connecting to 192.168.204.106@tcp115 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14513.982662] LNetError: Skipped 10 previous similar messages [14514.051429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14514.083850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14514.288354] Lustre: lustre-MDT0000: new disk, initializing [14514.417614] Lustre: 464690:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [14514.428117] Lustre: 464690:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14514.464110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14514.491854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14514.849991] LNet: 7 local NIs in recovery (showing 7): 192.168.204.106@tcp123, 192.168.204.106@tcp125, 192.168.204.106@tcp126, 192.168.204.106@tcp127, 192.168.204.106@tcp128, 192.168.204.106@tcp124, 192.168.204.106@tcp129 [14514.853391] LNetError: 462090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14514.875127] LNetError: 462090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14514.901092] LNetError: Protocol error connecting to 192.168.204.106@tcp129 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14514.920361] LNetError: Skipped 4 previous similar messages [14516.964387] LNetError: 464666:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14516.975677] LNetError: 464666:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14516.980247] LNetError: Protocol error connecting to 192.168.204.106@tcp129 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14519.181224] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14520.034500] LNetError: 464655:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14520.047956] LNetError: Protocol error connecting to 192.168.204.106@tcp129 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14528.227076] LNetError: 464660:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14528.234285] LNetError: Protocol error connecting to 192.168.204.106@tcp129 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14529.299245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14529.419177] Lustre: 464650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14529.494463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14529.501586] Lustre: Skipped 1 previous similar message [14529.613858] Lustre: lustre-MDT0001: new disk, initializing [14529.701160] Lustre: 465451:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [14529.711616] Lustre: 465451:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14529.932240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14530.012215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14530.029030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14536.424965] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14544.546124] LNetError: 462089:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14544.560733] LNetError: Protocol error connecting to 192.168.204.106@tcp121 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14547.559147] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14554.131652] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14561.958086] LNetError: 464656:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14561.972236] LNetError: 464656:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14561.994297] LNetError: Protocol error connecting to 192.168.204.106@tcp122 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14562.016397] LNetError: Skipped 11 previous similar messages [14562.156192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14562.684648] Lustre: lustre-OST0000: new disk, initializing [14562.698348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14562.715975] Lustre: 466625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14562.838468] Lustre: 466630:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [14562.852093] Lustre: 466630:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14562.929354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14570.039660] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14571.085573] Lustre: 467027:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [14571.104460] Lustre: 467027:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14571.164615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14571.173049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14571.331655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14574.305221] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp129 [14581.889879] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14591.969104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14591.975317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14591.983751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14591.997404] LustreError: Skipped 1 previous similar message [14592.011918] Lustre: Skipped 1 previous similar message [14597.105530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14597.125144] Lustre: Skipped 2 previous similar messages [14597.800595] LNetError: 462090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.106 [14597.813790] LNetError: 462090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [14597.833818] LNetError: Protocol error connecting to 192.168.204.106@tcp111 at host 192.168.204.106:988: is it running a compatible version of Lustre? [14597.846854] LNetError: Skipped 20 previous similar messages [14602.223035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14602.231054] Lustre: Skipped 1 previous similar message [14604.256187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14604.642613] Lustre: server umount lustre-OST0000 complete [14611.940499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14611.949476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14611.962055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14615.575971] Lustre: server umount lustre-MDT0000 complete [14619.472077] LustreError: 464644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787638623 with bad export cookie 9501787775874386129 [14619.486932] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14619.491520] LustreError: 464644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14619.802756] Lustre: server umount lustre-MDT0001 complete [14632.019885] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [14635.504942] Key type lgssc unregistered [14636.175445] LNet: 468543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14636.192130] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14636.203135] LNet: Removed LNI 192.168.204.106@tcp [14637.280414] LNet: Removed LNI 192.168.204.106@tcp100 [14637.286198] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14637.300045] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14637.319128] LNet: Removed LNI 192.168.204.106@tcp101 [14638.368243] LNet: Removed LNI 192.168.204.106@tcp102 [14638.374366] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14638.383604] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14638.388761] LNet: Removed LNI 192.168.204.106@tcp103 [14639.457284] LNet: Removed LNI 192.168.204.106@tcp104 [14640.480206] LNet: Removed LNI 192.168.204.106@tcp105 [14640.490000] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14640.511913] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14640.522772] LNet: Removed LNI 192.168.204.106@tcp106 [14641.568728] LNet: Removed LNI 192.168.204.106@tcp107 [14642.595840] LNet: Removed LNI 192.168.204.106@tcp108 [14643.616304] LNet: Removed LNI 192.168.204.106@tcp109 [14644.640802] LNet: Removed LNI 192.168.204.106@tcp110 [14644.645234] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14644.653965] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14644.670365] LNet: Removed LNI 192.168.204.106@tcp111 [14645.729039] LNet: Removed LNI 192.168.204.106@tcp112 [14646.752304] LNet: Removed LNI 192.168.204.106@tcp113 [14647.779387] LNet: Removed LNI 192.168.204.106@tcp114 [14648.804608] LNet: Removed LNI 192.168.204.106@tcp115 [14649.824592] LNet: Removed LNI 192.168.204.106@tcp116 [14650.848991] LNet: Removed LNI 192.168.204.106@tcp117 [14651.872151] LNet: Removed LNI 192.168.204.106@tcp118 [14652.896589] LNet: Removed LNI 192.168.204.106@tcp119 [14652.902620] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14652.926893] LNetError: 468543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14652.962171] LNet: Removed LNI 192.168.204.106@tcp120 [14654.048551] LNet: Removed LNI 192.168.204.106@tcp121 [14655.074155] LNet: Removed LNI 192.168.204.106@tcp122 [14656.096187] LNet: Removed LNI 192.168.204.106@tcp123 [14657.120736] LNet: Removed LNI 192.168.204.106@tcp124 [14658.144158] LNet: Removed LNI 192.168.204.106@tcp125 [14659.168190] LNet: Removed LNI 192.168.204.106@tcp126 [14660.192153] LNet: Removed LNI 192.168.204.106@tcp127 [14661.216201] LNet: Removed LNI 192.168.204.106@tcp128 [14661.224554] LNet: Removed LNI 192.168.204.106@tcp129 [14661.235593] LNet: Removed LNI 192.168.204.106@tcp130 [14661.243851] LNet: Removed LNI 192.168.204.106@tcp131 [14661.251857] LNet: Removed LNI 192.168.204.106@tcp132 [14661.257895] LNet: Removed LNI 192.168.204.106@tcp133 [14661.261181] LNet: Removed LNI 192.168.204.106@tcp134 [14661.272587] LNet: Removed LNI 192.168.204.106@tcp135 [14661.774410] Key type .llcrypt unregistered [14661.777264] Key type ._llcrypt unregistered [14697.262800] Key type ._llcrypt registered [14697.267684] Key type .llcrypt registered [14697.406097] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [14716.324577] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [14717.615403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14718.088719] alg: No test for adler32 (adler32-zlib) [14719.361636] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [14719.724591] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [14721.560172] Key type lgssc registered [14723.451788] Lustre: Echo OBD driver; http://www.lustre.org/ [14763.864739] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:19:25 (1787638765) [14785.684654] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [14789.930301] Key type lgssc unregistered [14790.374759] LNet: 472638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14790.383589] LNetError: 472638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14790.417675] LNet: Removed LNI 192.168.204.106@tcp [14791.687339] Key type .llcrypt unregistered [14791.690838] Key type ._llcrypt unregistered [14815.276032] Key type ._llcrypt registered [14815.281052] Key type .llcrypt registered [14815.504192] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [14817.891290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14817.963907] alg: No test for adler32 (adler32-zlib) [14819.196544] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [14819.484531] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [14821.176440] Key type lgssc registered [14822.327605] Lustre: Echo OBD driver; http://www.lustre.org/ [14836.005955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14836.023660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14837.455139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14837.564348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14837.740828] Lustre: lustre-MDT0000: new disk, initializing [14837.860417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14837.903871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14843.758667] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14857.280564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14857.538420] Lustre: 473854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14857.630831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14857.640917] Lustre: Skipped 1 previous similar message [14857.818398] Lustre: lustre-MDT0001: new disk, initializing [14858.009854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14858.091072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14858.128922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14864.062674] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14874.995942] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14882.224308] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14890.007624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14890.371366] Lustre: lustre-OST0000: new disk, initializing [14890.376441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14890.389453] Lustre: 475823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14890.513241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14897.820741] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [14899.271807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14899.284015] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14899.472554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14907.906910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14914.211401] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lnet_if_list [14921.086377] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14921.131247] LNet: Added LNI 192.168.204.106@tcp10 [8/256/0/180] [14928.706895] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14928.752022] LNet: Added LNI 192.168.204.106@tcp11 [8/256/0/180] [14933.737250] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14933.782360] LNet: Added LNI 192.168.204.106@tcp12 [8/256/0/180] [14939.524564] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14939.569220] LNet: Added LNI 192.168.204.106@tcp13 [8/256/0/180] [14945.494655] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14945.545848] LNet: Added LNI 192.168.204.106@tcp14 [8/256/0/180] [14950.511401] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14950.535376] LNet: Added LNI 192.168.204.106@tcp15 [8/256/0/180] [14955.741350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14955.777873] LNet: Added LNI 192.168.204.106@tcp16 [8/256/0/180] [14960.951975] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14961.005082] LNet: Added LNI 192.168.204.106@tcp17 [8/256/0/180] [14967.309241] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14967.352466] LNet: Added LNI 192.168.204.106@tcp18 [8/256/0/180] [14975.573573] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14975.604617] LNet: Added LNI 192.168.204.106@tcp19 [8/256/0/180] [14982.392608] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14982.439642] LNet: Added LNI 192.168.204.106@tcp20 [8/256/0/180] [14988.670630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14988.718726] LNet: Added LNI 192.168.204.106@tcp21 [8/256/0/180] [14994.332957] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14994.375265] LNet: Added LNI 192.168.204.106@tcp22 [8/256/0/180] [15000.246761] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15000.290910] LNet: Added LNI 192.168.204.106@tcp23 [8/256/0/180] [15006.374522] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15006.413978] LNet: Added LNI 192.168.204.106@tcp24 [8/256/0/180] [15014.896460] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15014.957514] LNet: Added LNI 192.168.204.106@tcp25 [8/256/0/180] [15022.660330] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15022.701838] LNet: Added LNI 192.168.204.106@tcp26 [8/256/0/180] [15029.093183] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15029.112976] LNet: Added LNI 192.168.204.106@tcp27 [8/256/0/180] [15034.211172] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15034.235501] LNet: Added LNI 192.168.204.106@tcp28 [8/256/0/180] [15040.572454] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15040.618202] LNet: Added LNI 192.168.204.106@tcp29 [8/256/0/180] [15047.674114] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15047.712568] LNet: Added LNI 192.168.204.106@tcp30 [8/256/0/180] [15054.707631] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15054.734064] LNet: Added LNI 192.168.204.106@tcp31 [8/256/0/180] [15062.040822] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15062.069293] LNet: Added LNI 192.168.204.106@tcp32 [8/256/0/180] [15068.959488] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15069.019281] LNet: Added LNI 192.168.204.106@tcp33 [8/256/0/180] [15076.602764] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15076.654886] LNet: Added LNI 192.168.204.106@tcp34 [8/256/0/180] [15085.525230] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15085.568692] LNet: Added LNI 192.168.204.106@tcp35 [8/256/0/180] [15096.344906] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15096.428469] LNet: Added LNI 192.168.204.106@tcp36 [8/256/0/180] [15106.953517] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15106.978277] LNet: Added LNI 192.168.204.106@tcp37 [8/256/0/180] [15115.794485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15115.816426] LNet: Added LNI 192.168.204.106@tcp38 [8/256/0/180] [15126.388052] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15126.440443] LNet: Added LNI 192.168.204.106@tcp39 [8/256/0/180] [15135.439386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15135.485060] LNet: Added LNI 192.168.204.106@tcp40 [8/256/0/180] [15145.115185] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15145.147317] LNet: Added LNI 192.168.204.106@tcp41 [8/256/0/180] [15155.386423] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15155.449559] LNet: Added LNI 192.168.204.106@tcp42 [8/256/0/180] [15167.049266] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15167.113666] LNet: Added LNI 192.168.204.106@tcp43 [8/256/0/180] [15175.679934] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15175.734915] LNet: Added LNI 192.168.204.106@tcp44 [8/256/0/180] [15184.312404] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15184.346441] LNet: Added LNI 192.168.204.106@tcp45 [8/256/0/180] [15194.406443] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15194.442863] LNet: Added LNI 192.168.204.106@tcp46 [8/256/0/180] [15205.014637] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15205.044602] LNet: Added LNI 192.168.204.106@tcp47 [8/256/0/180] [15214.409338] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15214.430814] LNet: Added LNI 192.168.204.106@tcp48 [8/256/0/180] [15224.102988] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15224.144364] LNet: Added LNI 192.168.204.106@tcp49 [8/256/0/180] [15234.597374] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15234.646838] LNet: Added LNI 192.168.204.106@tcp50 [8/256/0/180] [15256.043418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15256.057384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15256.074984] Lustre: Skipped 1 previous similar message [15256.103441] Lustre: Skipped 2 previous similar messages [15261.155422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15261.169175] Lustre: Skipped 2 previous similar messages [15266.276400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15266.291047] Lustre: Skipped 2 previous similar messages [15269.344131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15269.562578] Lustre: server umount lustre-MDT0000 complete [15272.869644] LustreError: 473846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787639276 with bad export cookie 2381199781532464665 [15272.875949] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15272.877547] LustreError: 473846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15273.233479] Lustre: server umount lustre-MDT0001 complete [15292.900775] Lustre: 473128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787639280/real 1787639280] req@000000007856ed2e x1874475166663168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787639296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15292.949982] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15292.972981] Lustre: Skipped 1 previous similar message [15293.736590] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [15293.923084] Lustre: 473127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787639281/real 1787639281] req@00000000d056011f x1874475166663296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787639297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15298.018309] Lustre: 473125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787639285/real 1787639285] req@0000000037da07d8 x1874475166663424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787639301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15301.412420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15303.136287] Lustre: 473128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787639291/real 1787639291] req@00000000dc825c45 x1874475166663808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787639307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15303.174792] Lustre: 473128:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15308.256168] Lustre: 473127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787639296/real 1787639296] req@00000000dfb5ea6b x1874475166663936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787639312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15317.613961] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x210bb895cd7b00b5 [15317.630636] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [15317.982101] LustreError: 475817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15318.015366] LustreError: 475817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15318.083762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15322.081309] LustreError: 475817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15322.107228] LustreError: 475817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15323.844602] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15326.185741] LustreError: 475818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15326.192378] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15331.308151] LustreError: 475817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15333.046343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15333.463676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15337.594994] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15338.703512] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15347.248987] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15353.411727] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15378.923755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15378.926923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15378.932077] Lustre: Skipped 1 previous similar message [15378.943683] Lustre: Skipped 1 previous similar message [15384.034029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15384.037184] Lustre: Skipped 1 previous similar message [15389.154766] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15389.410123] Lustre: server umount lustre-OST0000 complete [15394.274834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15394.277909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15394.298722] Lustre: Skipped 1 previous similar message [15394.309167] Lustre: Skipped 3 previous similar messages [15400.363335] Lustre: server umount lustre-MDT0000 complete [15404.516697] LustreError: 500080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15404.618925] LustreError: 500080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15406.410811] LustreError: 473845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787639410 with bad export cookie 2381199781532467381 [15406.416446] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15406.422515] LustreError: 473845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15406.912491] Lustre: server umount lustre-MDT0001 complete [15418.332332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [15420.890057] Key type lgssc unregistered [15421.327822] LNet: 502889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15421.335520] LNetError: 502889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15421.350891] LNet: Removed LNI 192.168.204.106@tcp [15421.352938] LNet: Removed LNI 192.168.204.106@tcp10 [15421.358878] LNet: Removed LNI 192.168.204.106@tcp11 [15421.366337] LNet: Removed LNI 192.168.204.106@tcp12 [15421.373200] LNet: Removed LNI 192.168.204.106@tcp13 [15421.378628] LNet: Removed LNI 192.168.204.106@tcp14 [15421.380865] LNet: Removed LNI 192.168.204.106@tcp15 [15421.383531] LNet: Removed LNI 192.168.204.106@tcp16 [15421.385821] LNet: Removed LNI 192.168.204.106@tcp17 [15421.390468] LNet: Removed LNI 192.168.204.106@tcp18 [15421.393592] LNet: Removed LNI 192.168.204.106@tcp19 [15421.397649] LNet: Removed LNI 192.168.204.106@tcp20 [15421.401376] LNet: Removed LNI 192.168.204.106@tcp21 [15421.405378] LNet: Removed LNI 192.168.204.106@tcp22 [15421.408777] LNet: Removed LNI 192.168.204.106@tcp23 [15421.411573] LNet: Removed LNI 192.168.204.106@tcp24 [15421.414748] LNet: Removed LNI 192.168.204.106@tcp25 [15421.418319] LNet: Removed LNI 192.168.204.106@tcp26 [15421.421175] LNet: Removed LNI 192.168.204.106@tcp27 [15421.424398] LNet: Removed LNI 192.168.204.106@tcp28 [15421.427439] LNet: Removed LNI 192.168.204.106@tcp29 [15421.430883] LNet: Removed LNI 192.168.204.106@tcp30 [15421.434303] LNet: Removed LNI 192.168.204.106@tcp31 [15421.436245] LNet: Removed LNI 192.168.204.106@tcp32 [15421.440470] LNet: Removed LNI 192.168.204.106@tcp33 [15421.442914] LNet: Removed LNI 192.168.204.106@tcp34 [15421.444470] LNet: Removed LNI 192.168.204.106@tcp35 [15421.445963] LNet: Removed LNI 192.168.204.106@tcp36 [15421.448307] LNet: Removed LNI 192.168.204.106@tcp37 [15421.450114] LNet: Removed LNI 192.168.204.106@tcp38 [15421.451662] LNet: Removed LNI 192.168.204.106@tcp39 [15421.454343] LNet: Removed LNI 192.168.204.106@tcp40 [15421.457315] LNet: Removed LNI 192.168.204.106@tcp41 [15421.459523] LNet: Removed LNI 192.168.204.106@tcp42 [15421.461668] LNet: Removed LNI 192.168.204.106@tcp43 [15421.463723] LNet: Removed LNI 192.168.204.106@tcp44 [15421.465144] LNet: Removed LNI 192.168.204.106@tcp45 [15421.468397] LNet: Removed LNI 192.168.204.106@tcp46 [15421.471176] LNet: Removed LNI 192.168.204.106@tcp47 [15421.473705] LNet: Removed LNI 192.168.204.106@tcp48 [15421.476648] LNet: Removed LNI 192.168.204.106@tcp49 [15421.497954] LNet: Removed LNI 192.168.204.106@tcp50 [15422.665324] Key type .llcrypt unregistered [15422.677494] Key type ._llcrypt unregistered [15444.628686] Key type ._llcrypt registered [15444.630047] Key type .llcrypt registered [15444.713424] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [15462.220701] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [15463.477043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15463.516357] alg: No test for adler32 (adler32-zlib) [15464.729564] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [15465.043390] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [15466.808175] Key type lgssc registered [15467.700131] Lustre: Echo OBD driver; http://www.lustre.org/ [15506.644802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15506.709761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15508.016498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15508.060291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15508.175292] Lustre: lustre-MDT0000: new disk, initializing [15508.287532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15508.322833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15514.015042] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15525.445865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15525.608298] Lustre: 506088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15525.711334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15525.717659] Lustre: Skipped 1 previous similar message [15525.794397] Lustre: lustre-MDT0001: new disk, initializing [15525.947537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15526.032318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15526.056071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15531.465127] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15541.862916] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15548.548362] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15556.941765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15557.110256] Lustre: *** cfs_fail_loc=90f, val=0*** [15557.163388] Lustre: lustre-OST0000: new disk, initializing [15557.166382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15557.171208] Lustre: 508103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15557.243806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15563.491625] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15563.864890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15563.881293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15563.967991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15572.865837] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15584.745321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15584.746332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15584.778053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15584.786587] Lustre: Skipped 1 previous similar message [15585.959811] Lustre: server umount lustre-OST0000 complete [15589.858668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15589.863491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15589.878271] Lustre: Skipped 2 previous similar messages [15589.893352] Lustre: Skipped 1 previous similar message [15594.980406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15594.989947] Lustre: Skipped 1 previous similar message [15595.530298] Lustre: server umount lustre-MDT0000 complete [15598.866944] LustreError: 508113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787639602 with bad export cookie 12623033379850356267 [15598.873806] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15598.875620] LustreError: 508113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15599.093661] Lustre: server umount lustre-MDT0001 complete [15610.551721] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [15614.069261] Key type lgssc unregistered [15614.679847] LNet: 510012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15614.697945] LNetError: 510012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15614.757379] LNet: Removed LNI 192.168.204.106@tcp [15616.053448] Key type .llcrypt unregistered [15616.056522] Key type ._llcrypt unregistered [15640.446341] Key type ._llcrypt registered [15640.447933] Key type .llcrypt registered [15640.584760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [15641.867528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15641.888051] alg: No test for adler32 (adler32-zlib) [15643.181293] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [15643.626583] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [15645.384175] Key type lgssc registered [15646.587834] Lustre: Echo OBD driver; http://www.lustre.org/ [15654.926805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15654.980218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15656.586024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15661.717515] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15666.169811] LustreError: 511286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15666.202315] LustreError: 511286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15671.035476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15671.389448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15676.542815] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15684.105043] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15689.819773] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15696.952766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15697.191628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15702.767703] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15709.718284] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15727.278655] loop0: detected capacity change from 0 to 204800000 [15727.484205] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15727.506825] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15727.537278] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15735.091240] loop0: detected capacity change from 0 to 204800000 [15735.398153] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15735.416868] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15735.428842] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15741.048694] loop0: detected capacity change from 0 to 204800000 [15747.205458] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15747.423653] Lustre: fs15246-OST0000: new disk, initializing [15747.434646] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15747.449487] Lustre: 514881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15747.535759] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15754.732373] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15765.024564] loop1: detected capacity change from 0 to 204800000 [15771.670587] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15771.809179] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15771.885439] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15771.993773] Lustre: fs15246-MDT0000: new disk, initializing [15772.219840] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15772.295544] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15772.479362] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15773.538994] Lustre: 516005:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bd73fedc x1874476030540160/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787639793 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15776.910365] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [15782.629380] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15782.640053] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15782.664306] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15801.834474] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15801.856676] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15804.489177] Lustre: server umount fs15246-MDT0000 complete [15806.969657] LustreError: 511282:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15812.072513] LustreError: 512828:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15815.969661] Lustre: server umount fs15246-OST0000 complete [15829.480088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15829.481098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15829.488666] LustreError: Skipped 1 previous similar message [15829.510551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15829.537715] Lustre: Skipped 1 previous similar message [15832.566641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15832.576042] Lustre: Skipped 1 previous similar message [15834.540971] Lustre: server umount lustre-OST0000 complete [15840.234850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15840.237185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15840.257872] Lustre: Skipped 1 previous similar message [15840.267288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15842.790085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15845.422838] Lustre: server umount lustre-MDT0000 complete [15847.910069] LustreError: 512013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15847.939035] LustreError: 512013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15850.319338] LustreError: 511267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787639854 with bad export cookie 16194967284143780123 [15850.324960] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15850.343047] LustreError: 511267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15850.870684] Lustre: server umount lustre-MDT0001 complete [15864.034232] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [15867.461527] Key type lgssc unregistered [15867.796551] LNet: 519106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15867.805102] LNetError: 519106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15867.823784] LNet: Removed LNI 192.168.204.106@tcp [15868.856847] Key type .llcrypt unregistered [15868.860500] Key type ._llcrypt unregistered [15892.847384] Key type ._llcrypt registered [15892.850158] Key type .llcrypt registered [15893.020847] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [15894.192781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15894.294520] alg: No test for adler32 (adler32-zlib) [15895.658875] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [15895.952762] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [15897.648438] Key type lgssc registered [15899.209874] Lustre: Echo OBD driver; http://www.lustre.org/ [15944.298861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [15955.836966] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [16000.631651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16000.675294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16002.178640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16002.258719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16002.390836] Lustre: lustre-MDT0000: new disk, initializing [16002.516720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16002.545220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16006.823270] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [16018.615662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16018.778432] Lustre: 523577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16018.863935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16018.870890] Lustre: Skipped 1 previous similar message [16018.969445] Lustre: lustre-MDT0001: new disk, initializing [16019.074024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16019.127321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16019.132878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16025.234649] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [16034.120695] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16039.615922] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16049.411945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16049.824699] Lustre: lustre-OST0000: new disk, initializing [16049.837887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16049.845518] Lustre: 525547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16049.954945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16056.935697] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [16057.936686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16057.949839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16058.101809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16067.818720] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16072.921160] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16073.225818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16077.039979] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16077.239396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16083.937558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16083.941845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16083.973821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16083.981564] Lustre: Skipped 1 previous similar message [16083.995865] Lustre: Skipped 1 previous similar message [16085.389527] Lustre: server umount lustre-OST0000 complete [16090.595123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16090.615157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16090.632934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16094.178517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16094.180037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16094.198389] Lustre: Skipped 1 previous similar message [16096.698974] Lustre: server umount lustre-MDT0000 complete [16099.312089] LustreError: 523588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16099.332858] LustreError: 523588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [16100.789346] LustreError: 523568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787640104 with bad export cookie 4829652939371236658 [16100.799026] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16100.809837] LustreError: 523568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16101.317953] Lustre: server umount lustre-MDT0001 complete [16133.601623] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [16143.830982] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [16176.122170] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15873 sec ========= 02:42:57 (1787640177) [16178.122748] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:42:59 (1787640179) === [16181.957636] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:43:03 (1787640183) === [16223.471146] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [16226.919219] Key type lgssc unregistered [16227.209162] LNet: 532196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16227.217841] LNetError: 532196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16228.262711] LNet: Removed LNI 192.168.204.106@tcp [16229.317295] Key type .llcrypt unregistered [16229.320206] Key type ._llcrypt unregistered