[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479586803 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003084] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013042] *** VALIDATE selinux *** [ 0.020766] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025219] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.031443] *** VALIDATE proc *** [ 0.032258] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036006] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042803] debug: unmapping init [mem 0xffffffff98259000-0xffffffff98260fff] [ 0.044159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053312] rcu: Hierarchical SRCU implementation. [ 0.055666] smp: Bringing up secondary CPUs ... [ 0.056587] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060703] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142116] node 0 deferred pages initialised in 70ms [ 0.146156] devtmpfs: initialized [ 0.148270] x86/mm: Memory block size: 128MB [ 0.150819] gcov: version magic: 0x41383552 [ 0.153312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159307] pinctrl core: initialized pinctrl subsystem [ 0.161206] [ 0.161669] ************************************************************* [ 0.163017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166062] ** ** [ 0.167009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169009] ** ** [ 0.171009] ** This means that this kernel is built to expose internal ** [ 0.173011] ** IOMMU data structures, which may compromise security on ** [ 0.174009] ** your system. ** [ 0.176010] ** ** [ 0.178009] ** If you see this message and you are not debugging the ** [ 0.180008] ** kernel, report this immediately to your vendor! ** [ 0.181007] ** ** [ 0.183009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185016] ************************************************************* [ 0.187502] NET: Registered protocol family 16 [ 0.190499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200475] cpuidle: using governor menu [ 0.201570] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204515] PCI: Using configuration type 1 for base access [ 0.207136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220290] cryptd: max_cpu_qlen set to 1000 [ 0.223300] ACPI: Added _OSI(Module Device) [ 0.226015] ACPI: Added _OSI(Processor Device) [ 0.228025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236636] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242806] ACPI: Interpreter enabled [ 0.245058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247011] ACPI: Using IOAPIC for interrupt routing [ 0.249096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267966] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287746] acpiphp: Slot [2] registered [ 0.291160] acpiphp: Slot [5] registered [ 0.294147] acpiphp: Slot [6] registered [ 0.298376] acpiphp: Slot [7] registered [ 0.300130] acpiphp: Slot [8] registered [ 0.302129] acpiphp: Slot [9] registered [ 0.304122] acpiphp: Slot [10] registered [ 0.306110] acpiphp: Slot [3] registered [ 0.308113] acpiphp: Slot [4] registered [ 0.311167] acpiphp: Slot [11] registered [ 0.312114] acpiphp: Slot [12] registered [ 0.314098] acpiphp: Slot [13] registered [ 0.316111] acpiphp: Slot [14] registered [ 0.317144] acpiphp: Slot [15] registered [ 0.319092] acpiphp: Slot [16] registered [ 0.321123] acpiphp: Slot [17] registered [ 0.323098] acpiphp: Slot [18] registered [ 0.324087] acpiphp: Slot [19] registered [ 0.326180] acpiphp: Slot [20] registered [ 0.327108] acpiphp: Slot [21] registered [ 0.329099] acpiphp: Slot [22] registered [ 0.331150] acpiphp: Slot [23] registered [ 0.332086] acpiphp: Slot [24] registered [ 0.334090] acpiphp: Slot [25] registered [ 0.336106] acpiphp: Slot [26] registered [ 0.338162] acpiphp: Slot [27] registered [ 0.340107] acpiphp: Slot [28] registered [ 0.342091] acpiphp: Slot [29] registered [ 0.343086] acpiphp: Slot [30] registered [ 0.345178] acpiphp: Slot [31] registered [ 0.347070] PCI host bridge to bus 0000:00 [ 0.349019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.352022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.361023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.364022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.366204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369099] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.373290] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382754] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388729] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.392019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.399017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402574] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.405914] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.411825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.431015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435957] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.479000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491519] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.507015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.519017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.539017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.553182] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.568024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.576014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.600014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.616881] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.627014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.638016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.661017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.670422] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.680015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.686011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.715783] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.723026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.753023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765733] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.769375] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.771378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.775491] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.777238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.782278] iommu: Default domain type: Passthrough [ 0.785798] SCSI subsystem initialized [ 0.788189] ACPI: bus type USB registered [ 0.791111] usbcore: registered new interface driver usbfs [ 0.794086] usbcore: registered new interface driver hub [ 0.797097] usbcore: registered new device driver usb [ 0.799161] pps_core: LinuxPPS API ver. 1 registered [ 0.801012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805090] PTP clock support registered [ 0.808045] EDAC MC: Ver: 3.0.0 [ 0.810134] PCI: Using ACPI for IRQ routing [ 0.811562] NetLabel: Initializing [ 0.813074] NetLabel: domain hash size = 128 [ 0.815028] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.817103] NetLabel: unlabeled traffic allowed by default [ 0.820372] vgaarb: loaded [ 0.821521] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.833119] clocksource: Switched to clocksource kvm-clock [ 0.940957] VFS: Disk quotas dquot_6.6.0 [ 0.942796] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.945687] *** VALIDATE ramfs *** [ 0.947303] *** VALIDATE hugetlbfs *** [ 0.949220] pnp: PnP ACPI init [ 0.951829] pnp: PnP ACPI: found 6 devices [ 0.968967] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.973031] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.976508] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.979775] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984188] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.988228] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.992574] NET: Registered protocol family 2 [ 0.996037] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.002241] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.006226] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.012507] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.016357] TCP: Hash tables configured (established 65536 bind 65536) [ 1.019566] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.023317] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026766] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030094] NET: Registered protocol family 1 [ 1.034202] RPC: Registered named UNIX socket transport module. [ 1.036314] RPC: Registered udp transport module. [ 1.038281] RPC: Registered tcp transport module. [ 1.040269] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.042863] NET: Registered protocol family 44 [ 1.044824] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.047324] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.049380] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051858] PCI: CLS 0 bytes, default 64 [ 1.054323] Unpacking initramfs... [ 2.436236] debug: unmapping init [mem 0xffff9506bcc54000-0xffff9506bffbffff] [ 2.440749] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.443623] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447336] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.946462] Initialise system trusted keyrings [ 2.948985] Key type blacklist registered [ 2.951504] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.962496] zbud: loaded [ 2.965695] *** VALIDATE nfs *** [ 2.967196] *** VALIDATE nfs4 *** [ 2.969443] pstore: using deflate compression [ 2.973130] Platform Keyring initialized [ 3.082695] NET: Registered protocol family 38 [ 3.084608] Key type asymmetric registered [ 3.085859] Asymmetric key parser 'x509' registered [ 3.087636] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.091251] io scheduler mq-deadline registered [ 3.093718] io scheduler kyber registered [ 3.095829] io scheduler bfq registered [ 3.097755] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.101133] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.104172] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.108097] ACPI: Power Button [PWRF] [ 3.114803] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.123872] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.145789] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.155403] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.174387] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.205069] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.234160] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.238248] Non-volatile memory driver v1.3 [ 3.239614] Linux agpgart interface v0.103 [ 3.271149] virtio_blk virtio1: [vda] 147224 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.275523] vda: detected capacity change from 0 to 75378688 [ 3.288479] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.291045] vdb: detected capacity change from 0 to 1073741824 [ 3.304832] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310079] vdc: detected capacity change from 0 to 2621440000 [ 3.326132] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.329146] vdd: detected capacity change from 0 to 2621440000 [ 3.342884] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.346108] vde: detected capacity change from 0 to 4294967296 [ 3.371268] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.375878] vdf: detected capacity change from 0 to 4294967296 [ 3.385658] libphy: Fixed MDIO Bus: probed [ 3.392670] usbcore: registered new interface driver usbserial_generic [ 3.395420] usbserial: USB Serial support registered for generic [ 3.398465] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.403858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.405948] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.408536] mousedev: PS/2 mouse device common for all mice [ 3.412897] rtc_cmos 00:05: RTC can wake from S4 [ 3.415715] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420646] rtc_cmos 00:05: registered as rtc0 [ 3.422753] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.427165] intel_pstate: CPU model not supported [ 3.428669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.434227] hid: raw HID events driver (C) Jiri Kosina [ 3.437116] usbcore: registered new interface driver usbhid [ 3.439584] usbhid: USB HID core driver [ 3.441154] drop_monitor: Initializing network drop monitor service [ 3.441181] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.443329] Initializing XFRM netlink socket [ 3.449369] NET: Registered protocol family 10 [ 3.452905] Segment Routing with IPv6 [ 3.454252] NET: Registered protocol family 17 [ 3.456263] mpls_gso: MPLS GSO support [ 3.462326] RAS: Correctable Errors collector initialized. [ 3.463734] AVX version of gcm_enc/dec engaged. [ 3.465231] AES CTR mode by8 optimization enabled [ 3.543624] sched_clock: Marking stable (3543603095, 0)->(4438937627, -895334532) [ 3.547109] registered taskstats version 1 [ 3.549117] Loading compiled-in X.509 certificates [ 3.551013] zswap: loaded using pool lzo/zbud [ 3.575583] Key type big_key registered [ 3.587991] Key type encrypted registered [ 3.589711] ima: No TPM chip found, activating TPM-bypass! [ 3.591931] ima: Allocated hash algorithm: sha1 [ 3.593841] ima: No architecture policies found [ 3.595667] evm: Initialising EVM extended attributes: [ 3.597726] evm: security.selinux [ 3.598946] evm: security.ima [ 3.600156] evm: security.capability [ 3.601509] evm: HMAC attrs: 0x1 [ 3.604330] rtc_cmos 00:05: setting system clock to 2026-09-11 11:39:04 UTC (1789126744) [ 3.610867] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.614085] debug: unmapping init [mem 0xffffffff97f82000-0xffffffff98258fff] [ 3.624077] Write protecting the kernel read-only data: 28672k [ 3.628166] debug: unmapping init [mem 0xffffffff96603000-0xffffffff967fffff] [ 3.631478] debug: unmapping init [mem 0xffffffff96f14000-0xffffffff96ffffff] [ 3.667673] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679018] systemd[1]: Detected virtualization kvm. [ 3.680824] systemd[1]: Detected architecture x86-64. [ 3.683063] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.708663] systemd[1]: No hostname configured. [ 3.710761] systemd[1]: Set hostname to . [ 3.712818] random: systemd: uninitialized urandom read (16 bytes read) [ 3.715522] systemd[1]: Initializing machine ID from random generator. [ 3.898333] random: systemd: uninitialized urandom read (16 bytes read) [ 3.901461] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.907189] random: systemd: uninitialized urandom read (16 bytes read) [ 3.910473] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.918631] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.689829] device-mapper: uevent: version 1.0.3 [ 4.692764] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.664038] virtio_net virtio0 ens2: renamed from eth0 [ 5.685570] random: fast init done [ 5.822737] scsi host0: ata_piix [ 5.861054] scsi host1: ata_piix [ 5.863823] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.866042] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.483951] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.107968] random: crng init done [ 10.109601] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.858970] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.391359] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.691710] SELinux: Disabled at runtime. [ 12.756951] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.766321] systemd[1]: Detected virtualization kvm. [ 12.768507] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.313499] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.317228] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.321040] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.325255] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.328996] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.336402] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.342271] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 13.487034] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.939540] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.371386] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.480159] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.622604] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.637591] EDAC sbridge: Ver: 1.1.2 [ 16.349671] Key type dns_resolver registered [ 16.700063] NFS: Registering the id_resolver key type [ 16.702539] Key type id_resolver registered [ 16.704293] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 54.290647] libcfs: loading out-of-tree module taints kernel. [ 54.354178] Key type ._llcrypt registered [ 54.356424] Key type .llcrypt registered [ 54.439545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 69.001228] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 70.341660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.352901] alg: No test for adler32 (adler32-zlib) [ 71.846362] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 72.785652] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 74.519290] Key type lgssc registered [ 76.019369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.024370] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.792414] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 140.822584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.842524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.013343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.047326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.125852] Lustre: lustre-MDT0000: new disk, initializing [ 142.199294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.222222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.020408] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.300481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.406395] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.435970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.441247] Lustre: Skipped 1 previous similar message [ 154.533158] Lustre: lustre-MDT0001: new disk, initializing [ 154.581243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.624016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.634043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.404798] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.204838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.817881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.043155] Lustre: lustre-OST0000: new disk, initializing [ 168.046473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.053689] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.103935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.364472] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.123061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.137789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 175.205127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.151711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.273377] Lustre: lustre-OST0001: new disk, initializing [ 183.277103] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.281790] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.351494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.385317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.395512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.432473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 188.022106] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.568816] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 203.477917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 207.043128] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 210.383622] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 213.936442] Lustre: DEBUG MARKER: Client: 2.17.57.137 [ 215.729299] Lustre: DEBUG MARKER: MDS: 2.17.57.137 [ 217.475661] Lustre: DEBUG MARKER: OSS: 2.17.57.137 [ 218.711166] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 07:42:38 EDT 2026 [ 231.910987] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 233.467509] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 250.852721] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.855232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.868969] Lustre: Skipped 2 previous similar messages [ 250.877801] Lustre: Skipped 1 previous similar message [ 254.516229] Lustre: server umount lustre-MDT0000 complete [ 257.232244] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126998 with bad export cookie 6282324987124137808 [ 257.232440] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 257.239443] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 261.088659] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.088904] LustreError: 6517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.089455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 261.089459] Lustre: Skipped 1 previous similar message [ 261.110498] Lustre: Skipped 2 previous similar messages [ 261.126118] LustreError: 6517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 262.111822] LustreError: 7681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.112308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 262.130922] Lustre: Skipped 2 previous similar messages [ 263.642368] Lustre: server umount lustre-MDT0001 complete [ 266.575044] Lustre: server umount lustre-OST0000 complete [ 270.072467] Lustre: server umount lustre-OST0001 complete [ 275.976452] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 284.504308] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 319.393971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 319.678976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 319.703699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 319.826521] Lustre: lustre-MDT0000: new disk, initializing [ 319.927782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 319.943716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 323.900711] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 332.212131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.329893] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 332.406855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 332.414535] Lustre: Skipped 1 previous similar message [ 332.511585] Lustre: lustre-MDT0001: new disk, initializing [ 332.603697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 332.631750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 332.640777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 336.326153] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 342.328654] hrtimer: interrupt took 1792955 ns [ 345.088917] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 350.952761] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 357.454817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 358.132428] Lustre: lustre-OST0000: new disk, initializing [ 358.140451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 358.153831] Lustre: 17031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 358.316342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.439126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 359.445570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 359.589188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 365.071528] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 372.587493] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 375.457447] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 375.586662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 378.514499] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 378.681316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 380.388193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 380.397252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.416985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 385.003470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 385.004597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.011291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 386.369612] Lustre: server umount lustre-OST0000 complete [ 393.190704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.199365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 393.212272] Lustre: Skipped 1 previous similar message [ 393.222904] Lustre: Skipped 2 previous similar messages [ 395.743963] Lustre: server umount lustre-MDT0000 complete [ 398.308418] LustreError: 15073:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.327594] LustreError: 15073:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 398.904553] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127139 with bad export cookie 6282324987124139341 [ 398.917716] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.924742] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 399.307648] Lustre: server umount lustre-MDT0001 complete [ 402.863650] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:45:42 (1789127142) [ 404.691906] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 406.054907] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:45:45 (1789127145) [ 411.205181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.618636] LustreError: 19441:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.679834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.802263] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 416.738918] LustreError: 19442:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.662736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.366959] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 432.078547] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.771652] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.112691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.138712] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 454.330393] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.912848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.914563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.918046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.935671] Lustre: Skipped 1 previous similar message [ 472.736896] Lustre: server umount lustre-OST0000 complete [ 482.275494] LustreError: 22041:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.291268] LustreError: 22041:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 482.513095] Lustre: server umount lustre-MDT0000 complete [ 487.172403] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127228 with bad export cookie 6282324987124140223 [ 487.175996] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.184233] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 487.664975] Lustre: server umount lustre-MDT0001 complete [ 498.454890] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 501.150445] Key type lgssc unregistered [ 501.543945] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.550323] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.578728] LNet: Removed LNI 192.168.203.106@tcp [ 502.493149] Key type .llcrypt unregistered [ 502.498728] Key type ._llcrypt unregistered [ 522.407276] Key type ._llcrypt registered [ 522.411579] Key type .llcrypt registered [ 522.499390] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 523.495503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.520042] alg: No test for adler32 (adler32-zlib) [ 524.674631] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 524.958438] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 526.639790] Key type lgssc registered [ 527.655710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 535.407692] loop: module loaded [ 535.467687] loop0: detected capacity change from 0 to 204800000 [ 535.649235] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.670313] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.938472] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.747365] loop0: detected capacity change from 0 to 204800000 [ 540.959180] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.977918] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.995712] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.056297] loop0: detected capacity change from 0 to 204800000 [ 549.470381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 549.498719] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.782789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 550.834473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 550.921692] Lustre: lustre-MDT0000: new disk, initializing [ 550.990013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.003791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 553.963901] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 561.626241] loop1: detected capacity change from 0 to 204800000 [ 567.376936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.617783] Lustre: lustre-OST07c6: new disk, initializing [ 567.628465] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 567.633212] Lustre: Skipped 1 previous similar message [ 567.637452] Lustre: 26181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 567.698045] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 574.506323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 577.145915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 577.155499] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 577.178680] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 579.993680] Lustre: 25169:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 580.008575] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 580.019939] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 580.032651] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 580.042562] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 580.047283] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 594.162368] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 594.169272] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 594.173742] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 594.179354] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 594.183539] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 594.188113] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 598.994159] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 599.024743] Lustre: 25168:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 599.033991] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 599.040896] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.049067] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 599.058638] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.065405] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 599.070365] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.075217] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 599.081401] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.086362] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.092484] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.256253] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 602.263158] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 602.271728] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 602.277253] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.286551] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 602.294526] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.310889] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 602.320639] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.329189] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 602.336019] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.342707] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.347731] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 606.826422] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 606.836948] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 606.844791] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 606.854161] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 606.865513] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 606.871260] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 606.876576] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 606.886444] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 606.894132] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 606.900563] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 606.908319] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 606.919500] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 616.418805] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 616.427672] Lustre: 25167:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 616.433311] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 616.437371] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 616.440601] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 616.445064] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 616.449798] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 616.455870] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 616.460913] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 616.469234] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 616.479977] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 616.487402] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.152149] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.158508] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.178825] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 674.278857] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 678.367258] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 678.479265] Lustre: server umount lustre-OST07c6 complete [ 685.630711] Lustre: server umount lustre-MDT0000 complete [ 697.221469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 699.337786] Key type lgssc unregistered [ 699.600827] LNet: 28739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.609369] LNetError: 28739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.648331] LNet: Removed LNI 192.168.203.106@tcp [ 701.286356] Key type .llcrypt unregistered [ 701.288710] Key type ._llcrypt unregistered [ 716.689458] Key type ._llcrypt registered [ 716.694556] Key type .llcrypt registered [ 716.794573] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 718.104221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.244981] alg: No test for adler32 (adler32-zlib) [ 719.313672] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 719.647780] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 721.344144] Key type lgssc registered [ 722.550344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.982566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.995728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.448455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.086871] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 741.860748] LustreError: 29988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.880070] LustreError: 29988:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 743.574783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.099476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 748.305225] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 754.635027] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 759.391722] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 764.523300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 764.811533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.104762] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 777.010884] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 777.142711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 789.472454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 789.479158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.497899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.544311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.550226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.569274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.577404] Lustre: Skipped 1 previous similar message [ 795.752651] Lustre: server umount lustre-OST0000 complete [ 800.743575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 800.757343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.783601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.785518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.798817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.822115] Lustre: Skipped 1 previous similar message [ 805.087540] Lustre: server umount lustre-MDT0000 complete [ 807.907668] LustreError: 29984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.932920] LustreError: 29984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 808.865550] LustreError: 29970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127549 with bad export cookie 4217936788062694107 [ 808.868599] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.875234] LustreError: 29970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 809.175516] Lustre: server umount lustre-MDT0001 complete [ 820.084355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 822.870451] Key type lgssc unregistered [ 823.168785] LNet: 33725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.175078] LNetError: 33725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.195979] LNet: Removed LNI 192.168.203.106@tcp [ 824.059357] Key type .llcrypt unregistered [ 824.061674] Key type ._llcrypt unregistered [ 841.472765] Key type ._llcrypt registered [ 841.476824] Key type .llcrypt registered [ 841.576822] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 842.680392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.730419] alg: No test for adler32 (adler32-zlib) [ 843.801805] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 844.074326] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 845.759212] Key type lgssc registered [ 846.668713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.435393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 853.463503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.863258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.335300] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 864.234673] LustreError: 35000:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.252302] LustreError: 35000:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 865.772288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.041255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.728186] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 875.172517] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 879.924898] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 885.932549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.224588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.726335] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 899.587518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 899.783713] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.816397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.830987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.850820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.912756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 914.922698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.924103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.961419] Lustre: Skipped 1 previous similar message [ 916.489964] Lustre: server umount lustre-OST0000 complete [ 922.592277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.598158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.618741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.163822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.164346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.393900] Lustre: server umount lustre-MDT0000 complete [ 929.396443] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127670 with bad export cookie 395016342327880656 [ 929.400599] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.409349] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 929.639231] Lustre: server umount lustre-MDT0001 complete [ 939.259203] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 941.740023] Key type lgssc unregistered [ 941.994832] LNet: 38537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.007682] LNetError: 38537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.032422] LNet: Removed LNI 192.168.203.106@tcp [ 942.726670] Key type .llcrypt unregistered [ 942.729444] Key type ._llcrypt unregistered [ 964.573440] Key type ._llcrypt registered [ 964.575464] Key type .llcrypt registered [ 964.661336] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 965.532814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.573715] alg: No test for adler32 (adler32-zlib) [ 966.682950] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 966.918533] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 968.599168] Key type lgssc registered [ 969.481515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.923413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.958421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.532529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.444906] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 988.146485] LustreError: 39953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.173827] LustreError: 39953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 991.869966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.330861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.901959] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1004.071286] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1009.884952] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1015.705960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.000736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.008127] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1023.165299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1028.570451] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1036.611537] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1036.849527] Lustre: server umount lustre-MDT0000 complete [ 1038.304303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1038.315050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.331850] LustreError: 39953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.818456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.830952] Lustre: Skipped 1 previous similar message [ 1039.928276] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127780 with bad export cookie 15288824628947953057 [ 1039.938884] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.708333] LustreError: 39948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.719209] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 1041.728172] LustreError: 39948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1043.935986] LustreError: 41908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.936991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.961495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1046.250170] Lustre: server umount lustre-MDT0001 complete [ 1066.532595] Lustre: server umount lustre-OST0000 complete [ 1080.439976] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1083.219635] Key type lgssc unregistered [ 1083.563455] LNet: 43815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.583592] LNetError: 43815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.648174] LNet: Removed LNI 192.168.203.106@tcp [ 1085.439551] Key type .llcrypt unregistered [ 1085.442802] Key type ._llcrypt unregistered [ 1104.757200] Key type ._llcrypt registered [ 1104.760226] Key type .llcrypt registered [ 1104.869381] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1105.636326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.717110] alg: No test for adler32 (adler32-zlib) [ 1106.805736] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1107.064579] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1108.719387] Key type lgssc registered [ 1109.647732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.266494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.287411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.656164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.350727] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1127.905940] LustreError: 45211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.918730] LustreError: 45211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1129.114273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.433266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1133.392541] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1139.931638] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1144.830812] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.992735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.205655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.392692] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1161.378050] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1162.547562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1167.845676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.852725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.853678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.861651] Lustre: Skipped 1 previous similar message [ 1167.877298] Lustre: Skipped 1 previous similar message [ 1169.225607] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1172.964352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1172.972775] Lustre: Skipped 1 previous similar message [ 1173.815869] Lustre: server umount lustre-OST0000 complete [ 1190.885510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.898604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.910226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.452508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.344272] Lustre: server umount lustre-MDT0000 complete [ 1197.601824] LustreError: 45210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.614654] LustreError: 45210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1199.086878] LustreError: 45193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127939 with bad export cookie 1426178735964735888 [ 1199.095279] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1199.097760] LustreError: 45193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1205.478292] Lustre: server umount lustre-MDT0001 complete [ 1215.139805] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1217.203623] Key type lgssc unregistered [ 1217.458164] LNet: 48923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.469389] LNetError: 48923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.481413] LNet: Removed LNI 192.168.203.106@tcp [ 1218.084599] Key type .llcrypt unregistered [ 1218.090150] Key type ._llcrypt unregistered [ 1236.623441] Key type ._llcrypt registered [ 1236.627025] Key type .llcrypt registered [ 1236.705126] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1237.597206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.641215] alg: No test for adler32 (adler32-zlib) [ 1238.667797] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1238.888629] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1240.543872] Key type lgssc registered [ 1241.615125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1248.071282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1248.085629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.525092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.174036] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1259.742418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.119708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.931159] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1273.282553] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.144496] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.987638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.256404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.642186] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1294.511526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1296.321055] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1310.954524] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1313.586994] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1326.428755] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-11 8h01m04s [ 1328.155876] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1330.661613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.673186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.693517] Lustre: Skipped 2 previous similar messages [ 1330.715020] Lustre: Skipped 2 previous similar messages [ 1331.749925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.765045] Lustre: Skipped 2 previous similar messages [ 1334.600597] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1336.988973] Lustre: server umount lustre-MDT0000 complete [ 1337.828514] LustreError: 50320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.862571] LustreError: 50320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1340.168582] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1341.734763] LustreError: 51308:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.981826] LustreError: 50316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.003255] LustreError: 50316:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1346.090949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.200306] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.532403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.632480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1347.570643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1350.936720] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1354.545921] Lustre: DEBUG MARKER: Wait for df (36689) ... [ 1356.167632] Lustre: DEBUG MARKER: done [ 1361.376735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.386928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.404084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1361.406803] Lustre: Skipped 1 previous similar message [ 1361.898534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1368.036837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.047315] Lustre: Skipped 3 previous similar messages [ 1374.175161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1374.508566] Lustre: server umount lustre-OST0000 complete [ 1383.396232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.405607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1383.405983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.405988] Lustre: Skipped 3 previous similar messages [ 1383.414937] Lustre: Skipped 2 previous similar messages [ 1393.121656] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1393.270975] Lustre: server umount lustre-MDT0000 complete [ 1396.327904] LustreError: 50300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128137 with bad export cookie 2312654931653223244 [ 1396.338193] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.536367] Lustre: server umount lustre-MDT0001 complete [ 1405.945152] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1408.751327] Key type lgssc unregistered [ 1409.034474] LNet: 55281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.047493] LNetError: 55281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.082771] LNet: Removed LNI 192.168.203.106@tcp [ 1409.764319] Key type .llcrypt unregistered [ 1409.768761] Key type ._llcrypt unregistered [ 1447.505940] Key type ._llcrypt registered [ 1447.507972] Key type .llcrypt registered [ 1447.620328] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1448.597549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.717400] alg: No test for adler32 (adler32-zlib) [ 1449.779686] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1450.009750] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1451.655170] Key type lgssc registered [ 1452.787845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1459.898535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1459.918244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.199214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1461.252059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1461.416292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.684714] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1473.282608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.368669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1473.421296] Lustre: 57490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1473.673795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.024289] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1485.475632] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1491.134632] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1497.464242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.610521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1497.774871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.081489] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1504.930747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1511.466839] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1517.425981] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-11 8h04m16s [ 1518.880411] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1525.408903] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1528.068585] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1528.108751] LustreError: 57496:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1548.649452] Lustre: lustre-MDT0000: Client f217322f-ff99-4e09-b7a5-860a8f49edbf (at 192.168.203.6@tcp) reconnecting [ 1568.207102] LustreError: 57496:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1570.988572] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1574.367050] Lustre: DEBUG MARKER: done [ 1581.538880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.545147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.563579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.048493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1582.065326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.085438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.096258] Lustre: Skipped 1 previous similar message [ 1587.173407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.176182] Lustre: Skipped 1 previous similar message [ 1592.304593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.312957] Lustre: Skipped 1 previous similar message [ 1594.335581] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.658991] Lustre: server umount lustre-OST0000 complete [ 1601.505057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1601.509532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.531855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.411606] Lustre: server umount lustre-MDT0000 complete [ 1607.870622] LustreError: 57480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128348 with bad export cookie 943825381287343661 [ 1607.880736] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1607.884066] LustreError: 57480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1608.159745] Lustre: server umount lustre-MDT0001 complete [ 1619.756934] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1622.408650] Key type lgssc unregistered [ 1622.696509] LNet: 61823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.715957] LNetError: 61823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.794837] LNet: Removed LNI 192.168.203.106@tcp [ 1624.804833] Key type .llcrypt unregistered [ 1624.809409] Key type ._llcrypt unregistered [ 1663.447240] Key type ._llcrypt registered [ 1663.449722] Key type .llcrypt registered [ 1663.566966] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1664.637227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.758046] alg: No test for adler32 (adler32-zlib) [ 1665.923202] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1666.221258] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1667.881947] Key type lgssc registered [ 1668.905584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1671.848484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1671.866346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.192131] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1673.209079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1673.439208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.252588] Lustre: Failing over lustre-MDT0000 [ 1674.464212] Lustre: server umount lustre-MDT0000 complete [ 1680.333126] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:06:59 (1789128419) [ 1686.912672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.384519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.304788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1698.933915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.054341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1699.082835] Lustre: 64609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1699.445053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1703.254757] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1710.237355] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1715.537453] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1722.378753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1722.646485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1722.894794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.860906] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1729.652292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1735.926769] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1741.445928] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1750.502210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1750.504597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1750.509507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1750.517300] Lustre: Skipped 2 previous similar messages [ 1750.524845] Lustre: Skipped 1 previous similar message [ 1753.508458] Lustre: server umount lustre-MDT0000 complete [ 1756.560073] LustreError: 64602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128497 with bad export cookie 10409858957552970815 [ 1756.563643] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.573994] LustreError: 64602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1756.911415] Lustre: server umount lustre-MDT0001 complete [ 1759.616942] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1763.316884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1771.254907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.119888] Lustre: 63533:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128502/real 1789128502] req@00000000b7db407b x1876037108539008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1777.123042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.158974] Lustre: 63533:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1781.215187] Lustre: 63535:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128506/real 1789128506] req@000000009394e928 x1876037108539136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1781.279675] LustreError: 63531:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002b45316e x1876037108540032/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1781.713934] LustreError: 66459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.739577] LustreError: 66459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1781.830094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.898493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1782.815156] Lustre: 63535:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128507/real 1789128507] req@0000000068ae6f0e x1876037108539392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1782.882018] LustreError: 66460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.894427] LustreError: 66459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.256518] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1787.873785] Lustre: 63533:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128512/real 1789128512] req@00000000aedce5f4 x1876037108539520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1787.937365] LustreError: 68378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.968389] LustreError: 68378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1792.995439] LustreError: 66460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.026389] LustreError: 66460:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1794.030757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.382627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.587629] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1799.652970] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1805.871244] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1806.455856] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.798368] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1820.134562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.159875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.181852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.191901] Lustre: Skipped 1 previous similar message [ 1824.495608] Lustre: server umount lustre-MDT0000 complete [ 1826.280226] LustreError: 69819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.293462] LustreError: 69819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1827.474220] LustreError: 65345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128568 with bad export cookie 10409858957552973090 [ 1827.478518] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.484962] LustreError: 65345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1827.745684] Lustre: server umount lustre-MDT0001 complete [ 1833.930895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1844.347532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.775315] Lustre: 63532:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128572/real 1789128572] req@00000000b249bd0a x1876037108567168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1847.806362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.822306] Lustre: Skipped 2 previous similar messages [ 1852.965817] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90773f916edcfa85 [ 1852.981151] Lustre: MGC192.168.203.106@tcp: Connection restored to 0@lo (at 0@lo) [ 1853.328714] LustreError: 66460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.417269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.499843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1858.463107] Lustre: 63532:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128583/real 1789128583] req@00000000ae8c4d8e x1876037108567808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789128599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1858.494000] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1858.498906] Lustre: 63532:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1866.954279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.666956] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1872.893910] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1877.039376] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1879.406254] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1885.910232] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1897.443857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.449642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.475895] Lustre: Skipped 2 previous similar messages [ 1897.491272] Lustre: Skipped 5 previous similar messages [ 1899.757854] Lustre: server umount lustre-MDT0000 complete [ 1902.585535] LustreError: 72124:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.611675] LustreError: 72124:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 1903.589934] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128644 with bad export cookie 10409858957552974469 [ 1903.593155] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1903.604685] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.005417] Lustre: server umount lustre-MDT0001 complete [ 1909.881915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1912.972138] Lustre: DEBUG MARKER: files compared the same [ 1924.428638] Lustre: server umount lustre-OST0000 complete [ 1943.750243] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1946.205362] Key type lgssc unregistered [ 1946.516206] LNet: 74353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.523648] LNetError: 74353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.543773] LNet: Removed LNI 192.168.203.106@tcp [ 1947.312369] Key type .llcrypt unregistered [ 1947.317176] Key type ._llcrypt unregistered [ 1970.655553] Key type ._llcrypt registered [ 1970.659955] Key type .llcrypt registered [ 1970.812514] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1972.324224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1972.338582] alg: No test for adler32 (adler32-zlib) [ 1973.443263] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 1973.793705] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1975.456226] Key type lgssc registered [ 1976.765594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.613416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1983.646365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.136364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.232484] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 1994.727614] LustreError: 75769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.739497] LustreError: 75769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1998.636609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.874594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.769737] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 2011.167383] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.581852] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2023.367035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.677304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.802281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2030.316791] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 2037.903806] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2049.504210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2049.509537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.513174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.520344] Lustre: Skipped 1 previous similar message [ 2050.916626] Lustre: server umount lustre-OST0000 complete [ 2054.624654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.626596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.634468] Lustre: Skipped 1 previous similar message [ 2054.643613] Lustre: Skipped 2 previous similar messages [ 2059.743889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.751835] Lustre: Skipped 1 previous similar message [ 2060.898480] Lustre: server umount lustre-MDT0000 complete [ 2064.836976] LustreError: 75751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128805 with bad export cookie 4348522830283691081 [ 2064.854166] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.233933] Lustre: server umount lustre-MDT0001 complete [ 2076.775464] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2079.776042] Key type lgssc unregistered [ 2080.135601] LNet: 79305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.146854] LNetError: 79305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.172771] LNet: Removed LNI 192.168.203.106@tcp [ 2081.184271] Key type .llcrypt unregistered [ 2081.189064] Key type ._llcrypt unregistered [ 2104.881311] Key type ._llcrypt registered [ 2104.886691] Key type .llcrypt registered [ 2105.083458] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2105.812686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.900623] alg: No test for adler32 (adler32-zlib) [ 2106.886185] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2107.058576] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2108.727211] Key type lgssc registered [ 2109.824348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2118.624622] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2118.666120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.239876] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2139.359303] LustreError: 80697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 2166.049846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.904932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2181.298368] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2189.212243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.347369] Lustre: *** cfs_fail_loc=706, val=0*** [ 2189.832852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.935097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2194.705580] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2194.930647] LustreError: 80723:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.963214] LustreError: 80723:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2200.046260] LustreError: 80722:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.076562] LustreError: 81784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.106105] LustreError: 81784:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2203.358109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.855853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.200995] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2217.569104] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2223.870391] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2229.730834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2229.738199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.745281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2229.780415] Lustre: Skipped 1 previous similar message [ 2232.414738] Lustre: server umount lustre-OST0000 complete [ 2239.978151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.983461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2239.990739] Lustre: Skipped 1 previous similar message [ 2239.991078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2240.026709] Lustre: Skipped 2 previous similar messages [ 2243.043425] Lustre: server umount lustre-MDT0000 complete [ 2245.095843] LustreError: 81785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.131279] LustreError: 81785:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2247.087101] LustreError: 80705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128987 with bad export cookie 5112919571292646964 [ 2247.102946] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.540919] Lustre: server umount lustre-MDT0001 complete [ 2259.475636] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2263.012224] Key type lgssc unregistered [ 2263.412154] LNet: 84321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.418258] LNetError: 84321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.489232] LNet: Removed LNI 192.168.203.106@tcp [ 2265.297608] Key type .llcrypt unregistered [ 2265.300609] Key type ._llcrypt unregistered [ 2293.144299] Key type ._llcrypt registered [ 2293.145888] Key type .llcrypt registered [ 2293.262464] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2294.122081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.167726] alg: No test for adler32 (adler32-zlib) [ 2295.252592] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2295.596450] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2297.304265] Key type lgssc registered [ 2298.226471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2307.048918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2307.061385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.834659] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2322.533082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.116847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.056577] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2328.561206] LustreError: 86429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.586604] LustreError: 86429:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2332.646299] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.664783] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2336.670806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.047799] LustreError: 86433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.074137] LustreError: 86433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2337.083318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.380098] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.401430] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2343.500165] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2347.493140] LustreError: 86429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.514521] LustreError: 86429:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2352.491377] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.515271] Lustre: MGS: Client 0ec4106b-a3e5-4605-95fa-742bcb984962 (at 0@lo) reconnecting [ 2352.539728] LustreError: 86443:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2352.614339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2352.619530] Lustre: Skipped 2 previous similar messages [ 2352.805920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.883246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2357.389541] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2373.605523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.608652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.623359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.662472] Lustre: Skipped 1 previous similar message [ 2376.678531] Lustre: server umount lustre-OST0000 complete [ 2383.840097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2383.843119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.858295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.873275] Lustre: Skipped 1 previous similar message [ 2383.881572] Lustre: Skipped 2 previous similar messages [ 2386.493099] Lustre: server umount lustre-MDT0000 complete [ 2388.961039] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.978263] LustreError: 86432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2390.245463] Lustre: server umount lustre-MDT0001 complete [ 2393.851035] Lustre: server umount MGS complete [ 2404.225598] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:19:03 (1789129143) [ 2431.303633] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2440.832436] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2476.659052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.679743] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2491.740431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.996290] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2492.087702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2492.212217] Lustre: lustre-MDT0001: new disk, initializing [ 2492.286172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.307816] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2492.316254] Lustre: 93596:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2492.335381] LustreError: 93596:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2497.002865] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2506.679578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2506.903226] Lustre: lustre-OST0000: new disk, initializing [ 2506.906365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2506.912991] Lustre: 94364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2506.972326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.345404] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2516.003232] Lustre: 94820:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d9ad6804 x1876037768092544/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789129272 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2521.524763] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2528.180205] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.202061] Lustre: MGS: Client 5880cb48-d3e4-4ca2-bbab-3536e6ff1dea (at 0@lo) reconnecting [ 2528.234694] LustreError: 93582:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2528.253680] Lustre: 92866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2528.290429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2528.399511] Lustre: lustre-MDT0000: new disk, initializing [ 2528.566345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.599480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2528.728820] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2529.636322] Lustre: 95433:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000501c33fe x1876037768099072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789129286 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2533.304570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2536.415673] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2538.799404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2538.813883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2538.886743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2563.559507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.566406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.580484] Lustre: Skipped 1 previous similar message [ 2568.721252] Lustre: server umount lustre-OST0000 complete [ 2573.793745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.797937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.803456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.803464] Lustre: Skipped 1 previous similar message [ 2573.813508] Lustre: Skipped 1 previous similar message [ 2578.683226] Lustre: server umount lustre-MDT0000 complete [ 2578.914603] LustreError: 95845:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.932732] LustreError: 95845:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2588.668969] Lustre: server umount lustre-MDT0001 complete [ 2592.303940] Lustre: server umount MGS complete [ 2602.060512] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:22:21 (1789129341) [ 2617.879436] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2620.243660] Key type lgssc unregistered [ 2620.457089] LNet: 97932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.463519] LNetError: 97932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.481593] LNet: Removed LNI 192.168.203.106@tcp [ 2621.256338] Key type .llcrypt unregistered [ 2621.259326] Key type ._llcrypt unregistered [ 2637.159217] Key type ._llcrypt registered [ 2637.161171] Key type .llcrypt registered [ 2637.261531] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2638.349484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.400681] alg: No test for adler32 (adler32-zlib) [ 2639.469759] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2639.715097] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2641.407164] Key type lgssc registered [ 2642.712739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.585765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.609797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.979681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.833847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2660.322825] LustreError: 99154:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.340849] LustreError: 99154:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2662.328281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.602173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.633897] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2672.795144] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2678.358544] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2684.667258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.062844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.322804] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2695.281730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2700.622751] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2709.985432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2710.002047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.026406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.041237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.042444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.059530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.808037] Lustre: server umount lustre-OST0000 complete [ 2721.249953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.253380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.276163] Lustre: Skipped 1 previous similar message [ 2721.296750] Lustre: Skipped 2 previous similar messages [ 2725.679773] Lustre: server umount lustre-MDT0000 complete [ 2726.369754] LustreError: 99155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.396462] LustreError: 99155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2729.672479] LustreError: 99135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129470 with bad export cookie 7023131493262905323 [ 2729.681196] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.701600] LustreError: 99135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2730.025620] Lustre: server umount lustre-MDT0001 complete [ 2740.766381] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2743.504246] Key type lgssc unregistered [ 2743.819562] LNet: 102689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2743.830628] LNetError: 102689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.846972] LNet: Removed LNI 192.168.203.106@tcp [ 2744.698303] Key type .llcrypt unregistered [ 2744.700021] Key type ._llcrypt unregistered [ 2759.416960] Key type ._llcrypt registered [ 2759.421549] Key type .llcrypt registered [ 2759.517781] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2760.472323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.577665] alg: No test for adler32 (adler32-zlib) [ 2761.634248] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2761.814096] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2763.455151] Key type lgssc registered [ 2764.647359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2770.470497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2770.480666] LustreError: 103767:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2770.515573] LustreError: 103771:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2770.520730] LustreError: 103767:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2770.529837] LustreError: 103767:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2770.535953] LustreError: 103767:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2770.540911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.987772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.695189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.931964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.535329] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2789.410580] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lsmod [ 2792.320803] LustreError: 104974:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2792.340755] LustreError: 104977:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2792.351610] LustreError: 104974:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 2792.359962] LustreError: 104974:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2792.374617] LustreError: 104974:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2792.378527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2792.796643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.050400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.055865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2798.060541] Lustre: Skipped 2 previous similar messages [ 2798.074865] Lustre: Skipped 2 previous similar messages [ 2801.938961] Lustre: server umount lustre-MDT0000 complete [ 2803.170199] LustreError: 105013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.193277] LustreError: 105013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2805.506967] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2805.507140] LustreError: 105016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129546 with bad export cookie 16956206588743128847 [ 2805.532887] LustreError: 105016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2807.327327] LustreError: 104239:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002f682b4 x1876038256708352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2807.474739] Lustre: server umount lustre-MDT0001 complete [ 2821.176198] Lustre: server umount lustre-OST0000 complete [ 2828.974884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.336461] LustreError: 106042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.401689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.678808] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2834.402239] LustreError: 106043:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.520233] LustreError: 106042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.959935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.299114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.341582] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2853.419976] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2858.385470] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2866.169863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.640686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.264269] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2875.844935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2881.514257] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2888.162644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2888.180235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.204555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.744213] Lustre: server umount lustre-OST0000 complete [ 2893.794219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.799835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.827316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.360128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.361412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.480974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.485946] Lustre: Skipped 2 previous similar messages [ 2903.564050] Lustre: server umount lustre-MDT0000 complete [ 2906.595798] LustreError: 106042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.611573] LustreError: 106042:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2906.705193] LustreError: 106022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129647 with bad export cookie 16956206588743129533 [ 2906.706693] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.718239] LustreError: 106022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2906.963770] Lustre: server umount lustre-MDT0001 complete [ 2916.765831] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2919.441770] Key type lgssc unregistered [ 2919.731694] LNet: 109581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.748248] LNetError: 109581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.813089] LNet: Removed LNI 192.168.203.106@tcp [ 2921.696250] Key type .llcrypt unregistered [ 2921.698212] Key type ._llcrypt unregistered [ 2946.030196] Key type ._llcrypt registered [ 2946.032781] Key type .llcrypt registered [ 2946.130583] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2947.159317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2947.304877] alg: No test for adler32 (adler32-zlib) [ 2948.417656] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 2948.680133] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2950.352273] Key type lgssc registered [ 2951.423557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.718676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.738315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.180983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.726847] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2970.605387] LustreError: 110999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.634032] LustreError: 110999:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2975.413548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.715037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2975.853806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.545849] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2987.200570] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2992.408716] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2998.772694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.161182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.215329] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3005.301356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3012.464379] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3018.109585] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3021.738135] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3031.015919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3031.019360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.043476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.063719] Lustre: Skipped 1 previous similar message [ 3031.078974] Lustre: Skipped 1 previous similar message [ 3032.768789] Lustre: server umount lustre-OST0000 complete [ 3037.172215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3037.195454] LustreError: Skipped 1 previous similar message [ 3037.205855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.225894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.249260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.252025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.148143] Lustre: server umount lustre-MDT0000 complete [ 3045.968356] LustreError: 113886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129786 with bad export cookie 1591277163165874721 [ 3045.977822] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.979458] LustreError: 113886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3046.229569] Lustre: server umount lustre-MDT0001 complete [ 3055.100883] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3058.029625] Key type lgssc unregistered [ 3058.355894] LNet: 114641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3058.370884] LNetError: 114641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3058.394502] LNet: Removed LNI 192.168.203.106@tcp [ 3059.199656] Key type .llcrypt unregistered [ 3059.201811] Key type ._llcrypt unregistered [ 3074.545845] Key type ._llcrypt registered [ 3074.547860] Key type .llcrypt registered [ 3074.663403] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3075.572743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3075.592514] alg: No test for adler32 (adler32-zlib) [ 3076.557745] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3076.760942] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3078.439215] Key type lgssc registered [ 3079.469974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3087.174246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3087.201788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.690749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.305145] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3099.109380] LustreError: 115859:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.147581] LustreError: 115859:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3101.305806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.596222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.994931] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3112.107281] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3117.563857] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3124.212705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.648579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.447358] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3134.914387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3138.036692] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3147.232671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3147.240157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.251258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.314451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.316538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.341987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.543963] Lustre: server umount lustre-OST0000 complete [ 3157.993145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3158.007053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.028038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3158.036344] Lustre: Skipped 1 previous similar message [ 3160.549085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.555171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.661844] Lustre: server umount lustre-MDT0000 complete [ 3165.668126] LustreError: 115854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.685855] LustreError: 115854:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3166.464299] LustreError: 115840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129907 with bad export cookie 15675768126546076293 [ 3166.475257] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.893188] Lustre: server umount lustre-MDT0001 complete [ 3177.584682] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3180.484186] Key type lgssc unregistered [ 3180.785644] LNet: 119393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.800647] LNetError: 119393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.825908] LNet: Removed LNI 192.168.203.106@tcp [ 3181.521232] Key type .llcrypt unregistered [ 3181.523312] Key type ._llcrypt unregistered [ 3208.387256] Key type ._llcrypt registered [ 3208.395890] Key type .llcrypt registered [ 3208.531651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3209.791790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.835375] alg: No test for adler32 (adler32-zlib) [ 3211.010738] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3211.208589] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3212.887411] Key type lgssc registered [ 3214.498858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3222.888222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3222.915696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.562471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.308498] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3234.794952] LustreError: 120810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.820142] LustreError: 120810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3236.478934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.860437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.944364] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3247.596331] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3252.617804] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3257.997852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3258.320043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.923388] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3269.097838] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3269.622940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3275.894663] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3279.106749] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3279.114490] Lustre: Skipped 2 previous similar messages [ 3282.675604] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3282.681985] Lustre: Skipped 2 previous similar messages [ 3283.535585] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3283.572477] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3289.009503] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3289.024987] Lustre: Skipped 2 previous similar messages [ 3291.113166] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp [ 3295.928461] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.6/24@tcp in log lustre-MDT0000 [ 3295.936241] Lustre: Skipped 2 previous similar messages [ 3300.008819] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 3300.017582] Lustre: Skipped 1 previous similar message [ 3305.449556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.451784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.454121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.471146] Lustre: Skipped 1 previous similar message [ 3310.560242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3310.574144] Lustre: Skipped 2 previous similar messages [ 3311.254942] Lustre: server umount lustre-OST0000 complete [ 3315.689929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.698778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.712110] Lustre: Skipped 1 previous similar message [ 3320.430967] Lustre: server umount lustre-MDT0000 complete [ 3320.807059] LustreError: 122057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.820163] LustreError: 122057:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3323.983285] LustreError: 123440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130064 with bad export cookie 8054049760732951351 [ 3323.987370] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3323.995276] LustreError: 123440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3324.366456] Lustre: server umount lustre-MDT0001 complete [ 3333.874485] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3337.002357] Key type lgssc unregistered [ 3337.311966] LNet: 125554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.323928] LNetError: 125554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.354186] LNet: Removed LNI 192.168.203.106@tcp [ 3338.546659] Key type .llcrypt unregistered [ 3338.550883] Key type ._llcrypt unregistered [ 3371.208346] Key type ._llcrypt registered [ 3371.213267] Key type .llcrypt registered [ 3371.314251] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3372.192709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3372.228772] alg: No test for adler32 (adler32-zlib) [ 3373.263738] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3373.509448] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3375.167293] Key type lgssc registered [ 3376.159927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3382.708153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3382.747528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.233721] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3384.262608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.267524] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.6/24@tcp [ 3389.462076] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3394.528701] LustreError: 127117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.538787] LustreError: 127117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3396.456181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.662982] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3396.677279] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 3396.702811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.792962] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3408.704402] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3413.485885] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.469568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.850058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.700482] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3430.065518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3431.460448] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.613276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.617339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.619803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.628537] Lustre: Skipped 1 previous similar message [ 3445.731805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.740846] Lustre: Skipped 2 previous similar messages [ 3447.014880] Lustre: server umount lustre-OST0000 complete [ 3450.850604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.860410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3450.874511] Lustre: Skipped 1 previous similar message [ 3450.885844] Lustre: Skipped 1 previous similar message [ 3455.968348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.976746] Lustre: Skipped 1 previous similar message [ 3456.947730] Lustre: server umount lustre-MDT0000 complete [ 3460.857372] LustreError: 127096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130201 with bad export cookie 6607007888891888111 [ 3460.872457] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.278319] Lustre: server umount lustre-MDT0001 complete [ 3472.625938] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3475.849202] Key type lgssc unregistered [ 3476.216742] LNet: 130706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.226684] LNetError: 130706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.248151] LNet: Removed LNI 192.168.203.106@tcp [ 3477.201595] Key type .llcrypt unregistered [ 3477.203684] Key type ._llcrypt unregistered [ 3503.831507] Key type ._llcrypt registered [ 3503.834957] Key type .llcrypt registered [ 3503.994106] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3519.659783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3520.788282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3520.891648] alg: No test for adler32 (adler32-zlib) [ 3521.845692] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3522.036400] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3523.687446] Key type lgssc registered [ 3524.734571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3556.227541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3556.263958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.490527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.519642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3557.628635] Lustre: lustre-MDT0000: new disk, initializing [ 3557.734911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.763592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3562.308112] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3572.417918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.556345] Lustre: 134109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.622345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3572.628471] Lustre: Skipped 1 previous similar message [ 3572.713336] Lustre: lustre-MDT0001: new disk, initializing [ 3572.868587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3572.955681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3572.974402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.905441] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3586.684036] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3591.684098] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3597.448105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.674386] Lustre: lustre-OST0000: new disk, initializing [ 3597.680220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3597.687885] Lustre: 136078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3597.861290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.175212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3604.602954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3604.620723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3604.757614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3613.925501] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.420879] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3620.743501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3626.440315] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3626.641493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3630.561436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.561795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.571267] LustreError: Skipped 1 previous similar message [ 3630.586221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.795802] Lustre: server umount lustre-OST0000 complete [ 3639.271454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3639.283733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.306327] Lustre: Skipped 1 previous similar message [ 3639.317325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.320776] Lustre: Skipped 1 previous similar message [ 3640.809757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.810660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.268562] Lustre: server umount lustre-MDT0000 complete [ 3647.677163] LustreError: 134100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130388 with bad export cookie 12117506980579089756 [ 3647.693254] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.694777] LustreError: 134100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3647.944288] Lustre: server umount lustre-MDT0001 complete [ 3656.917051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.632878] LustreError: 138291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3657.661609] LustreError: 138291:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3657.814995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.818685] LustreError: 138292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.144363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3667.942393] LustreError: 138291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.247995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.691792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3676.942684] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3684.929295] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3690.337381] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3696.309255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3696.678271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.106154] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3710.038339] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3714.436543] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3714.580805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3718.721890] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3718.928276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3728.024402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3728.228112] Lustre: lustre-OST0001: new disk, initializing [ 3728.235984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3728.239735] Lustre: lustre-OST0001: Not available for connect from 192.168.203.6@tcp (not set up) [ 3728.249346] Lustre: Skipped 1 previous similar message [ 3728.250533] Lustre: 141661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3728.370459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3734.095936] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3736.648820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3736.685790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3736.814791] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3742.172376] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3742.396454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3747.505629] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3747.686607] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3753.018647] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9afec3587000.ost_server_uuid 50 [ 3754.384309] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9afec3587000.ost_server_uuid in FULL state after 0 sec [ 3762.663990] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.664737] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.666598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3762.683158] Lustre: Skipped 1 previous similar message [ 3767.041592] Lustre: server umount lustre-OST0001 complete [ 3767.783452] LustreError: 140112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.800182] LustreError: 140112:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3772.896857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.914235] Lustre: Skipped 1 previous similar message [ 3772.944985] LustreError: 140113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.953843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3772.985364] LustreError: 140113:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3773.000686] Lustre: Skipped 2 previous similar messages [ 3776.763947] Lustre: server umount lustre-OST0000 complete [ 3783.144360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.178394] Lustre: Skipped 1 previous similar message [ 3787.572440] Lustre: server umount lustre-MDT0000 complete [ 3788.276991] LustreError: 138291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.312965] LustreError: 138291:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3792.532049] LustreError: 138273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130533 with bad export cookie 12117506980579090638 [ 3792.539392] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.563326] LustreError: 138273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3793.258088] Lustre: server umount lustre-MDT0001 complete [ 3815.582334] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3818.466859] Key type lgssc unregistered [ 3818.746738] LNet: 144667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3818.757070] LNetError: 144667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3818.783206] LNet: Removed LNI 192.168.203.106@tcp [ 3819.853775] Key type .llcrypt unregistered [ 3819.855755] Key type ._llcrypt unregistered [ 3858.441576] Key type ._llcrypt registered [ 3858.444501] Key type .llcrypt registered [ 3858.526175] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3874.321819] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3875.830274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.892403] alg: No test for adler32 (adler32-zlib) [ 3876.994328] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 3877.290075] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3879.015192] Key type lgssc registered [ 3880.502845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3915.961893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3916.001644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.397088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3917.424459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3917.532250] Lustre: lustre-MDT0000: new disk, initializing [ 3917.649991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.685203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3923.115230] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3934.112870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.271372] Lustre: 148827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3934.343392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3934.349191] Lustre: Skipped 1 previous similar message [ 3934.424630] Lustre: lustre-MDT0001: new disk, initializing [ 3934.500496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.534659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3934.547962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3938.218823] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3946.612860] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3952.058721] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3959.340385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.714514] Lustre: lustre-OST0000: new disk, initializing [ 3959.719714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3959.728203] Lustre: 150795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3959.862472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3966.312188] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3968.614520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3968.624499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3968.731387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3975.999968] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3980.965925] Lustre: Failing over lustre-OST0000 [ 3981.163299] Lustre: server umount lustre-OST0000 complete [ 3998.689131] Lustre: 147047:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130723/real 1789130723] req@000000006d1cb726 x1876039426933632/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789130739 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3998.715510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.957591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.241821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.259527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.272505] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4000.287440] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130725/real 1789130725] req@000000002851ea9a x1876039426933888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789130741 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4000.328179] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4003.300434] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4005.347894] Lustre: 147048:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130730/real 1789130730] req@000000000b6576c7 x1876039426935680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789130746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4005.389620] Lustre: 147048:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4005.613300] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4008.524365] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4008.524576] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4009.371935] Lustre: Failing over lustre-MDT0000 [ 4009.777428] Lustre: server umount lustre-MDT0000 complete [ 4010.402610] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130735/real 1789130735] req@00000000b4fe4b93 x1876039426936832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789130751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4011.494778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4011.503253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.523552] LustreError: 148837:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.541744] LustreError: 148837:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4013.536383] LustreError: 149636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.537510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.564357] LustreError: 149636:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4013.580520] Lustre: Skipped 1 previous similar message [ 4014.543200] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130739/real 1789130739] req@000000005e4e16ee x1876039426937984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789130755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4018.659866] LustreError: 148838:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.696608] LustreError: 148838:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4023.778115] LustreError: 148837:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.793412] LustreError: 148837:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4028.898486] LustreError: 148834:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.920234] LustreError: 148834:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4029.189886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.330353] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.631656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.702686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4029.740096] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4034.573362] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4035.058848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4035.063935] Lustre: Skipped 1 previous similar message [ 4035.079052] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4035.139967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4044.769857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4044.782054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.795976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4048.729208] Lustre: server umount lustre-OST0000 complete [ 4055.520635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.535680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.540639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4055.540647] Lustre: Skipped 2 previous similar messages [ 4055.579151] Lustre: Skipped 2 previous similar messages [ 4058.426101] Lustre: server umount lustre-MDT0000 complete [ 4060.640316] LustreError: 148838:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.665618] LustreError: 148838:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4062.040809] LustreError: 149630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130802 with bad export cookie 14964206134523036358 [ 4062.053160] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.462927] Lustre: server umount lustre-MDT0001 complete [ 4074.256728] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4077.379774] Key type lgssc unregistered [ 4077.636758] LNet: 154453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4077.644433] LNetError: 154453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4077.665088] LNet: Removed LNI 192.168.203.106@tcp [ 4078.517726] Key type .llcrypt unregistered [ 4078.520406] Key type ._llcrypt unregistered [ 4113.230474] Key type ._llcrypt registered [ 4113.232619] Key type .llcrypt registered [ 4113.356864] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4114.445698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4114.468419] alg: No test for adler32 (adler32-zlib) [ 4115.428271] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 4115.624572] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4117.285227] Key type lgssc registered [ 4118.122764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4125.647069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4125.674435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.236914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.625680] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4137.453153] LustreError: 155975:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.464187] LustreError: 155975:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4142.428240] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4142.573112] LustreError: 155974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.514381] LustreError: 156003:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4146.536820] LustreError: 156003:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4146.550791] LustreError: 156003:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4146.981593] Lustre: server umount lustre-MDT0000 complete [ 4163.895380] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4173.627878] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4207.018747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.273670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4207.320447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4207.411238] Lustre: lustre-MDT0000: new disk, initializing [ 4207.494991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.513465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4211.998113] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4222.116045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.236239] Lustre: 159306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4222.287926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4222.292457] Lustre: Skipped 1 previous similar message [ 4222.355853] Lustre: lustre-MDT0001: new disk, initializing [ 4222.462995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4222.498864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4222.513393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4227.113844] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4235.735265] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4241.208801] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4246.470312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4246.749499] Lustre: lustre-OST0000: new disk, initializing [ 4246.754562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4246.763762] Lustre: 161275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4246.850521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.493596] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4255.301813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4255.311750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4255.505986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4261.535560] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4266.219552] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4267.642088] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:50:06 (1789131006) [ 4491.233760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.237932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.251932] Lustre: Skipped 1 previous similar message [ 4495.127476] Lustre: server umount lustre-MDT0000 complete [ 4496.359398] LustreError: 159318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.381886] LustreError: 159318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4498.467289] LustreError: 159300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131239 with bad export cookie 11930304893591297441 [ 4498.470840] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.488503] LustreError: 159300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.471732] LustreError: 159847:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.471792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.518334] Lustre: Skipped 1 previous similar message [ 4501.533720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4501.549233] Lustre: Skipped 2 previous similar messages [ 4503.520571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4505.143540] Lustre: server umount lustre-MDT0001 complete [ 4515.325794] Lustre: server umount lustre-OST0000 complete [ 4524.999387] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4534.918518] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4569.090659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.361583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.399701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.473778] Lustre: lustre-MDT0000: new disk, initializing [ 4569.547403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.578839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.970501] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4583.381986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.491884] Lustre: 165932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4583.543656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4583.548873] Lustre: Skipped 1 previous similar message [ 4583.622262] Lustre: lustre-MDT0001: new disk, initializing [ 4583.748519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.782320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4583.790325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4587.769659] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4595.948326] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4601.126780] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.958180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.191053] Lustre: lustre-OST0000: new disk, initializing [ 4607.194329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4607.203646] Lustre: 167901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4607.300117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.827783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4617.361406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4617.369229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4617.424553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4620.581806] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4624.786639] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4624.954645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4629.194861] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4629.395304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4633.063896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4633.073556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.079941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4633.080643] Lustre: Skipped 1 previous similar message [ 4633.095956] Lustre: Skipped 1 previous similar message [ 4637.424376] Lustre: server umount lustre-OST0000 complete [ 4643.302474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.306596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.312669] Lustre: Skipped 1 previous similar message [ 4643.322251] Lustre: Skipped 1 previous similar message [ 4647.084435] Lustre: server umount lustre-MDT0000 complete [ 4648.418799] LustreError: 165939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.444952] LustreError: 165939:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4650.574258] LustreError: 165926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131391 with bad export cookie 11930304893591456264 [ 4650.583651] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4650.836651] Lustre: server umount lustre-MDT0001 complete [ 4657.883430] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:56:37 (1789131397) [ 4674.536668] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4686.760083] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4718.934885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.153519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4719.211226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4719.318450] Lustre: lustre-MDT0000: new disk, initializing [ 4719.400345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4719.418474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4723.969331] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4733.871635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.998415] Lustre: 172658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4734.288977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4737.813677] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4745.533833] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4750.587428] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4756.079207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.324625] Lustre: lustre-OST0000: new disk, initializing [ 4756.328633] Lustre: Skipped 1 previous similar message [ 4756.334742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4756.339758] Lustre: Skipped 2 previous similar messages [ 4756.344620] Lustre: 174635:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4761.868225] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4766.269560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4766.279207] Lustre: Skipped 1 previous similar message [ 4766.281283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4766.397414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4770.076386] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4776.934776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.936970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4776.941467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4776.949756] Lustre: Skipped 1 previous similar message [ 4776.970960] LustreError: Skipped 1 previous similar message [ 4781.828858] Lustre: server umount lustre-OST0000 complete [ 4785.636609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4785.641323] LustreError: Skipped 1 previous similar message [ 4785.648500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.182298] Lustre: server umount lustre-MDT0000 complete [ 4792.288475] LustreError: 172671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.306519] LustreError: 172671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4794.611982] LustreError: 172650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131535 with bad export cookie 11930304893591457153 [ 4794.616420] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.624030] LustreError: 172650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4794.887415] Lustre: server umount lustre-MDT0001 complete [ 4802.545275] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:59:01 (1789131541) [ 4818.345367] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4828.955690] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4862.347929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.642879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4862.697140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4862.776268] Lustre: lustre-MDT0000: new disk, initializing [ 4862.876188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.881676] Lustre: Skipped 2 previous similar messages [ 4862.899984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4866.854317] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4876.037679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.205815] Lustre: 178956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4876.535570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4880.594336] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4889.395797] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4895.393851] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4901.286359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4901.531658] Lustre: 180938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4907.182196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4907.712854] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4916.300430] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4922.851372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.854820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.864468] Lustre: Skipped 1 previous similar message [ 4922.865608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.865614] Lustre: Skipped 4 previous similar messages [ 4924.279872] Lustre: server umount lustre-OST0000 complete [ 4927.969894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4933.777171] Lustre: server umount lustre-MDT0000 complete [ 4936.679616] LustreError: 179767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131677 with bad export cookie 11930304893591458238 [ 4936.691452] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.693097] LustreError: 179767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4937.104724] Lustre: server umount lustre-MDT0001 complete [ 4947.197302] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4949.790251] Key type lgssc unregistered [ 4950.076599] LNet: 182806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4950.083601] LNetError: 182806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4950.103523] LNet: Removed LNI 192.168.203.106@tcp [ 4950.965591] Key type .llcrypt unregistered [ 4950.970284] Key type ._llcrypt unregistered [ 4973.400759] Key type ._llcrypt registered [ 4973.404895] Key type .llcrypt registered [ 4973.519947] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4974.195490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4974.210704] alg: No test for adler32 (adler32-zlib) [ 4975.245916] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 4975.472517] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4977.151161] Key type lgssc registered [ 4978.282952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4985.212279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4985.233602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.687961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.489145] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4997.095017] LustreError: 184223:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.112887] LustreError: 184223:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4999.482376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.871499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.160672] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5011.437327] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5016.540455] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5022.226280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.559470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.852429] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5031.796278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5034.241561] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5052.398057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.401235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.410946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.438549] Lustre: Skipped 1 previous similar message [ 5057.506868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5057.511866] Lustre: Skipped 2 previous similar messages [ 5058.521890] Lustre: server umount lustre-OST0000 complete [ 5062.624992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.638509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.646931] Lustre: Skipped 1 previous similar message [ 5062.657616] Lustre: Skipped 1 previous similar message [ 5067.747391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.756849] Lustre: Skipped 1 previous similar message [ 5068.401201] Lustre: server umount lustre-MDT0000 complete [ 5072.048516] LustreError: 184204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131812 with bad export cookie 13812315256152246100 [ 5072.078124] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.337731] Lustre: server umount lustre-MDT0001 complete [ 5083.288511] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5086.073625] Key type lgssc unregistered [ 5086.267900] LNet: 187777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5086.273690] LNetError: 187777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5086.306491] LNet: Removed LNI 192.168.203.106@tcp [ 5087.202158] Key type .llcrypt unregistered [ 5087.204481] Key type ._llcrypt unregistered [ 5108.264174] Key type ._llcrypt registered [ 5108.276320] Key type .llcrypt registered [ 5108.420505] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5109.445266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5109.513051] alg: No test for adler32 (adler32-zlib) [ 5110.652137] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 5110.978281] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5112.655172] Key type lgssc registered [ 5113.925285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5122.146886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5122.170568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.671175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.096894] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5133.795943] LustreError: 189175:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.814424] LustreError: 189175:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5136.345830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.771158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.890276] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5147.626804] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5153.912646] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5161.319323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5161.784248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.825939] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5168.946956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5174.602653] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5182.787257] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 5183.457260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5183.464325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.481855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5184.486464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5184.495825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.507566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5184.509862] Lustre: Skipped 1 previous similar message [ 5187.885303] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 5193.001774] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 5193.008161] Lustre: Skipped 2 previous similar messages [ 5195.231244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5195.436857] Lustre: server umount lustre-OST0000 complete [ 5202.291033] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9afec4b2f800.ost_server_uuid 50 [ 5204.019738] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5218.785039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5218.789359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.797890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5218.803042] Lustre: Skipped 2 previous similar messages [ 5224.106952] Lustre: server umount lustre-MDT0000 complete [ 5224.416899] LustreError: 190464:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.439595] LustreError: 190464:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5227.185787] LustreError: 189154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131968 with bad export cookie 16906277614012191276 [ 5227.192454] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.195796] LustreError: 189154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5227.491251] Lustre: server umount lustre-MDT0001 complete [ 5234.026739] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:06:13 (1789131973) [ 5239.541074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.967761] LustreError: 193092:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.050363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.493366] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5245.411971] LustreError: 193092:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.428785] LustreError: 193092:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5249.516969] LustreError: 193093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.716881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.107309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5256.512376] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5264.812091] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5270.206815] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5276.044762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5276.415953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.320754] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5284.606898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5291.938763] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5299.015661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5299.282749] Lustre: lustre-OST0001: new disk, initializing [ 5299.299445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5299.312645] Lustre: 195932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5299.406891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5305.442189] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5309.038369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5309.060023] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5309.123290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5314.660932] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5318.632362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.648915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.668753] Lustre: Skipped 1 previous similar message [ 5318.677734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.686634] Lustre: Skipped 2 previous similar messages [ 5323.712395] Lustre: server umount lustre-OST0000 complete [ 5324.775380] LustreError: 194914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.782661] LustreError: 194914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5331.717225] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5331.920619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5337.461193] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5337.750533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5340.991698] LustreError: 194913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.039182] LustreError: 194913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5355.488432] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5355.488470] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.503608] LustreError: Skipped 1 previous similar message [ 5355.505879] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5355.521863] Lustre: Skipped 2 previous similar messages [ 5355.525133] Lustre: Skipped 3 previous similar messages [ 5358.044489] Lustre: server umount lustre-OST0001 complete [ 5364.705499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5364.709083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.338955] Lustre: server umount lustre-MDT0000 complete [ 5371.949472] LustreError: 195931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132112 with bad export cookie 16906277614012192347 [ 5371.951670] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5371.959906] LustreError: 195931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5372.336862] Lustre: server umount lustre-MDT0001 complete [ 5396.336122] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:08:55 (1789132135) [ 5403.089332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.501538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5403.536555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5403.846890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.400509] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5416.555036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.720749] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5416.781207] Lustre: 199373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5422.226992] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5429.791887] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5435.592090] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5442.235184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5442.483301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5448.979762] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5449.367466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5456.475950] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5462.763945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5462.835667] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5468.679916] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5469.815259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5475.351820] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5479.906739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5479.914168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.930778] Lustre: Skipped 1 previous similar message [ 5479.945891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.954505] Lustre: Skipped 4 previous similar messages [ 5484.262949] Lustre: server umount lustre-OST0000 complete [ 5485.555337] LustreError: 201221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.572662] LustreError: 201221:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 5505.288984] Lustre: server umount lustre-OST0001 complete [ 5515.186313] Lustre: server umount lustre-MDT0000 complete [ 5518.560512] LustreError: 199365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132259 with bad export cookie 16906277614012193754 [ 5518.575058] LustreError: 199365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5518.578247] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.937194] Lustre: server umount lustre-MDT0001 complete [ 5540.258774] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:11:19 (1789132279) [ 5555.306298] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5564.849873] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5595.995047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5596.251756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5596.283278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5596.374421] Lustre: lustre-MDT0000: new disk, initializing [ 5596.455591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.464846] Lustre: Skipped 3 previous similar messages [ 5596.479896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5601.255642] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5610.684862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.811965] Lustre: 207463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5610.892827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5610.905912] Lustre: Skipped 1 previous similar message [ 5611.045402] Lustre: lustre-MDT0001: new disk, initializing [ 5611.238651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5611.247834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5615.372926] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5623.559661] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5629.224912] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5637.162674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5637.391806] Lustre: lustre-OST0000: new disk, initializing [ 5637.398301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5637.404516] Lustre: 209428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5643.380906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5643.414349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5643.558873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5644.464573] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5653.813472] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5657.939835] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5658.221050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5662.524211] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5662.744398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5669.360397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5669.360569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.366390] LustreError: Skipped 4 previous similar messages [ 5669.367367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5669.390136] Lustre: Skipped 6 previous similar messages [ 5669.412976] Lustre: Skipped 9 previous similar messages [ 5679.071101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5679.230186] Lustre: server umount lustre-OST0000 complete [ 5688.880129] Lustre: server umount lustre-MDT0000 complete [ 5689.825991] LustreError: 207474:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.843595] LustreError: 207474:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 5692.689282] LustreError: 207454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132433 with bad export cookie 16906277614012195532 [ 5692.698415] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5692.710989] LustreError: 207454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5693.138352] Lustre: server umount lustre-MDT0001 complete [ 5701.286610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.755788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5713.408400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.807630] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5725.754515] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5731.374603] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5737.592841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.879952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.886271] Lustre: Skipped 4 previous similar messages [ 5743.821618] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5751.141830] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5756.555491] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5756.764034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5761.173634] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5761.365376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5765.600185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.604950] LustreError: Skipped 2 previous similar messages [ 5765.608756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.622287] Lustre: Skipped 2 previous similar messages [ 5777.375143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5777.537823] Lustre: server umount lustre-OST0000 complete [ 5783.735628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5788.462240] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5817.143652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.501625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.519542] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5824.037767] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5827.626861] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.645864] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5831.473266] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5836.343877] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5855.199175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5855.405585] Lustre: server umount lustre-OST0000 complete [ 5873.119214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5873.470848] Lustre: server umount lustre-MDT0000 complete [ 5873.644320] LustreError: 212644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.674127] LustreError: 212644:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5877.304270] LustreError: 211628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132618 with bad export cookie 16906277614012196421 [ 5877.306850] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5877.327272] LustreError: 211628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5877.872758] Lustre: server umount lustre-MDT0001 complete [ 5886.356223] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:17:05 (1789132625) [ 5892.673072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.697172] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5906.777394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.033062] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5919.701232] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5926.024312] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5932.500069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.613364] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5947.128511] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5952.015993] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5952.221953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5957.121610] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5957.332996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5964.395817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.605525] Lustre: lustre-OST0001: new disk, initializing [ 5964.608775] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5964.614353] Lustre: 220989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5970.558124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5970.567794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5970.631920] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5970.665079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5980.035394] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5985.273807] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5985.496598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5990.485809] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5990.712972] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5996.525510] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.529495] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.530910] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5996.530917] Lustre: Skipped 21 previous similar messages [ 5996.544050] Lustre: Skipped 6 previous similar messages [ 5996.568914] LustreError: Skipped 5 previous similar messages [ 5998.641129] Lustre: server umount lustre-OST0001 complete [ 6004.872389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6005.082604] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6009.576491] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6009.779703] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6038.499342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.666776] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.676063] Lustre: Skipped 5 previous similar messages [ 6044.160748] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6044.172690] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6044.175742] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6044.497944] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6051.063482] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6052.470714] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6060.740502] Lustre: server umount lustre-OST0001 complete [ 6092.255154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.444865] Lustre: server umount lustre-OST0000 complete [ 6102.905443] Lustre: server umount lustre-MDT0000 complete [ 6106.201690] LustreError: 217601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132847 with bad export cookie 16906277614012197387 [ 6106.202338] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6106.207953] LustreError: 217601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6112.564662] Lustre: server umount lustre-MDT0001 complete [ 6135.201594] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:21:14 (1789132874) [ 6141.466361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.741952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6141.799594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6146.842451] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6155.277355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.369779] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6155.405237] Lustre: 226233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6159.766870] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6166.592256] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6171.970892] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6178.247495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.415050] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6184.426789] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6191.912338] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6201.037886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6201.107046] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6207.975401] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6215.744897] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6217.245359] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6222.036345] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6222.225257] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6226.628738] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6226.874405] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6232.040667] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9afec488c000.ost_server_uuid 50 [ 6233.718835] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9afec488c000.ost_server_uuid in IDLE state after 0 sec [ 6234.585254] Lustre: Permanently deactivating lustre-OST0001 [ 6234.618527] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6238.447399] Lustre: Permanently reactivating lustre-OST0001 [ 6246.128468] Lustre: server umount lustre-OST0001 complete [ 6247.392415] LustreError: 229394:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.427146] LustreError: 229394:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 42 previous similar messages [ 6256.489611] Lustre: server umount lustre-OST0000 complete [ 6262.752193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.779523] Lustre: Skipped 10 previous similar messages [ 6266.725163] Lustre: server umount lustre-MDT0000 complete [ 6271.005379] LustreError: 226225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133011 with bad export cookie 16906277614012198675 [ 6271.013076] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.017592] LustreError: 226225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6271.550130] Lustre: server umount lustre-MDT0001 complete [ 6296.146849] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:23:54 (1789133034) [ 6303.937216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.399236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6304.464989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6304.493559] Lustre: Skipped 5 previous similar messages [ 6309.495743] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6317.107987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.223826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6317.254981] Lustre: 232924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6322.183215] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6329.510595] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6335.689481] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6342.963510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6343.145570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6343.165340] Lustre: Permanently deactivating lustre-OST0000 [ 6343.177873] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6350.096544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6351.919768] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6357.155209] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6363.618766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.692484] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6369.640300] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6370.990432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6377.353235] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6379.178451] Lustre: Permanently reactivating lustre-OST0000 [ 6379.208747] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6379.214268] Lustre: Skipped 2 previous similar messages [ 6380.422105] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6380.429728] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.438978] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6380.454378] Lustre: Skipped 1 previous similar message [ 6380.494088] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6380.510870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6391.781185] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.798316] LustreError: Skipped 4 previous similar messages [ 6392.190166] Lustre: server umount lustre-OST0001 complete [ 6401.780644] Lustre: server umount lustre-OST0000 complete [ 6411.650396] Lustre: server umount lustre-MDT0000 complete [ 6415.239409] LustreError: 232917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133156 with bad export cookie 16906277614012200355 [ 6415.240109] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.249155] LustreError: 232917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6415.624517] Lustre: server umount lustre-MDT0001 complete [ 6426.647570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6429.646875] Key type lgssc unregistered [ 6429.903809] LNet: 237787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6429.915633] LNetError: 237787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6429.940374] LNet: Removed LNI 192.168.203.106@tcp [ 6430.849654] Key type .llcrypt unregistered [ 6430.854131] Key type ._llcrypt unregistered [ 6453.233197] Key type ._llcrypt registered [ 6453.244734] Key type .llcrypt registered [ 6453.394889] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6454.673868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6454.697759] alg: No test for adler32 (adler32-zlib) [ 6455.700208] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6455.949639] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6457.663168] Key type lgssc registered [ 6458.913242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6468.777296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6468.803336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.326674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6475.193477] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6480.870985] LustreError: 239255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.894219] LustreError: 239255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6483.001855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.107731] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6483.144897] Lustre: 239244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.153835] Lustre: Permanently deactivating lustre-MDT0001 [ 6483.171480] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6483.232362] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6483.241775] LustreError: 239285:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6483.259721] LustreError: 239285:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6483.280191] LustreError: 239285:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 6483.496743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.407676] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6493.694306] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6498.610337] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6504.959032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.219369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.326400] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.339898] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6509.959770] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6513.646575] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.693269] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.715104] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6515.716525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6516.923076] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.807297] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.819786] LustreError: 241095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6523.185711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.366631] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6528.510924] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6528.534713] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6529.192187] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6536.298638] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6539.992672] Lustre: Permanently reactivating lustre-MDT0001 [ 6540.011897] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6540.022427] Lustre: Skipped 2 previous similar messages [ 6541.765315] Lustre: lustre-MDT0001: Client 652681cf-74fa-4ef1-b0ca-07692b88ad3d (at 192.168.203.6@tcp) reconnecting [ 6547.856523] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6549.602580] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6555.017981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6555.205941] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6558.832746] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6559.051970] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6561.626990] Lustre: Permanently deactivating lustre-MDT0001 [ 6561.644948] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6561.649990] Lustre: Skipped 2 previous similar messages [ 6569.006775] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6575.587731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.588821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6575.606040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.635394] Lustre: Skipped 3 previous similar messages [ 6577.490673] Lustre: server umount lustre-MDT0000 complete [ 6580.704903] LustreError: 239255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.732955] LustreError: 239255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6581.598143] LustreError: 239235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133322 with bad export cookie 11454197611566584771 [ 6581.617656] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6582.009396] Lustre: server umount lustre-MDT0001 complete [ 6587.593815] Lustre: server umount lustre-OST0000 complete [ 6598.561328] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:28:57 (1789133337) [ 6608.303476] Lustre: server umount lustre-OST0001 complete [ 6613.617170] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 6622.921363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6654.797087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6655.014350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6655.021544] Lustre: Skipped 2 previous similar messages [ 6655.036117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6655.115897] Lustre: lustre-MDT0000: new disk, initializing [ 6655.230149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.246717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6659.246288] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6670.038866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.178924] Lustre: 247509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6670.249633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6670.256713] Lustre: Skipped 1 previous similar message [ 6670.319317] Lustre: lustre-MDT0001: new disk, initializing [ 6670.427981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6670.531189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6670.553948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6677.216655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6686.775972] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6693.085914] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6699.689163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6700.013081] Lustre: lustre-OST0000: new disk, initializing [ 6700.021051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6700.026340] Lustre: 249478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6700.181695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6707.083629] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6708.828287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6708.844281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6708.923416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6716.872798] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6719.201863] LustreError: 247514:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6727.803508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.973799] Lustre: lustre-OST0001: new disk, initializing [ 6727.977899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6727.983885] Lustre: 250591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6729.257555] LustreError: 247514:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6729.288593] LustreError: 249334:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6732.880506] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6735.942105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6735.950736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6736.024473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6739.384413] LustreError: 249334:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6741.928867] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6743.487228] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6746.598554] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.610979] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6746.618270] Lustre: Skipped 4 previous similar messages [ 6746.627368] Lustre: Skipped 1 previous similar message [ 6751.601195] Lustre: server umount lustre-OST0001 complete [ 6751.722229] LustreError: 249461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.748706] LustreError: 249461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6770.656550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.664827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.667237] LustreError: Skipped 1 previous similar message [ 6770.707216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.716589] Lustre: Skipped 1 previous similar message [ 6774.533712] Lustre: server umount lustre-OST0000 complete [ 6782.445627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.462795] Lustre: Skipped 1 previous similar message [ 6782.477216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.490642] Lustre: Skipped 3 previous similar messages [ 6784.379304] Lustre: server umount lustre-MDT0000 complete [ 6787.555592] LustreError: 249334:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.586100] LustreError: 249334:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6787.834024] LustreError: 247500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133528 with bad export cookie 11454197611566587585 [ 6787.837589] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.845920] LustreError: 247500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6802.400827] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6802.762388] Lustre: server umount lustre-MDT0001 complete [ 6812.768729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6815.833893] Key type lgssc unregistered [ 6816.236765] LNet: 252628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6816.252996] LNetError: 252628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6816.283637] LNet: Removed LNI 192.168.203.106@tcp [ 6817.408575] Key type .llcrypt unregistered [ 6817.417984] Key type ._llcrypt unregistered [ 6856.692824] Key type ._llcrypt registered [ 6856.700598] Key type .llcrypt registered [ 6856.841289] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6858.101228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6858.140917] alg: No test for adler32 (adler32-zlib) [ 6859.250182] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 6859.586730] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6861.320709] Key type lgssc registered [ 6862.421453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6871.009021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6871.029264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.544149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6872.574484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6872.884031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.823122] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6886.649110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.761520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6886.795050] Lustre: 254842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6886.949393] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6887.171317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6892.523673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6900.863353] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.195585] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6914.209839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6914.427930] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6914.676206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.738596] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6924.929558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6927.595101] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.600558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.608963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.616404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.227364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.236582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.292969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.174444] Lustre: server umount lustre-OST0000 complete [ 6993.670883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7007.984317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7008.491308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.518646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7011.557385] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7016.427076] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7025.054178] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7026.929066] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7026.952130] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7068.130822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7068.158033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.180286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.195204] Lustre: Skipped 2 previous similar messages [ 7072.116457] Lustre: server umount lustre-OST0000 complete [ 7076.833711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7076.839088] LustreError: Skipped 1 previous similar message [ 7076.840929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.848163] Lustre: Skipped 1 previous similar message [ 7076.852584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.608798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.621126] Lustre: Skipped 5 previous similar messages [ 7091.167290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7091.483956] Lustre: server umount lustre-MDT0000 complete [ 7095.337321] LustreError: 254832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133836 with bad export cookie 7758996667857552671 [ 7095.342047] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7095.350859] LustreError: 254832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7095.676417] Lustre: server umount lustre-MDT0001 complete [ 7106.444945] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7108.905829] Key type lgssc unregistered [ 7109.198292] LNet: 260280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7109.209845] LNetError: 260280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7109.244919] LNet: Removed LNI 192.168.203.106@tcp [ 7110.129247] Key type .llcrypt unregistered [ 7110.131545] Key type ._llcrypt unregistered [ 7135.315387] Key type ._llcrypt registered [ 7135.320790] Key type .llcrypt registered [ 7135.431208] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7137.288448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7137.363559] alg: No test for adler32 (adler32-zlib) [ 7138.543601] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7138.881552] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7140.647185] Key type lgssc registered [ 7142.081403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7149.760685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7149.781529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.194104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.719521] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7161.325549] LustreError: 261695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.343523] LustreError: 261695:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7164.186486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.650829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.026431] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7176.554834] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7181.859509] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7188.476282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.847679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.005810] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7197.041889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7202.625639] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7225.824509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.841384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.852162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7227.875899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7227.879901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.881417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.907552] Lustre: server umount lustre-OST0000 complete [ 7236.576949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7236.590276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.605183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.615987] Lustre: Skipped 1 previous similar message [ 7240.050277] Lustre: server umount lustre-MDT0000 complete [ 7243.237068] LustreError: 261696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.276129] LustreError: 261696:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7244.186438] LustreError: 261677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133985 with bad export cookie 10597722767646773539 [ 7244.200529] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7244.206545] LustreError: 261677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7244.797889] Lustre: server umount lustre-MDT0001 complete [ 7256.008750] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7259.057580] Key type lgssc unregistered [ 7259.379031] LNet: 266005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7259.390572] LNetError: 266005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7260.456661] LNet: Removed LNI 192.168.203.106@tcp [ 7261.544641] Key type .llcrypt unregistered [ 7261.552401] Key type ._llcrypt unregistered [ 7280.408567] Key type ._llcrypt registered [ 7280.414419] Key type .llcrypt registered [ 7280.516827] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7281.491161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7281.508600] alg: No test for adler32 (adler32-zlib) [ 7282.570356] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7282.780224] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7284.471233] Key type lgssc registered [ 7285.737357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7293.364784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7293.412454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.918091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.011788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7304.162282] LustreError: 267229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.184652] LustreError: 267229:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7309.298188] LustreError: 267230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.998107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.320473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7317.428556] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7324.905806] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7330.868648] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.037202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.384295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.440855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7342.669290] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7348.873554] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7358.944980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7358.951646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.980810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.970307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.974632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7360.010097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7360.015990] Lustre: Skipped 1 previous similar message [ 7364.773801] Lustre: server umount lustre-OST0000 complete [ 7370.248962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.325737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.339558] Lustre: Skipped 1 previous similar message [ 7370.358652] Lustre: Skipped 1 previous similar message [ 7375.328464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7375.337881] Lustre: Skipped 1 previous similar message [ 7376.363902] Lustre: server umount lustre-MDT0000 complete [ 7380.450224] LustreError: 267958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.481774] LustreError: 267958:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7380.682818] LustreError: 267211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134121 with bad export cookie 8627951865937779191 [ 7380.687065] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.693764] LustreError: 267211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7381.565353] Lustre: server umount lustre-MDT0001 complete [ 7390.681360] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7394.002229] Key type lgssc unregistered [ 7394.394867] LNet: 270911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7394.410447] LNetError: 270911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.433748] LNet: Removed LNI 192.168.203.106@tcp [ 7395.399196] Key type .llcrypt unregistered [ 7395.402801] Key type ._llcrypt unregistered [ 7417.603712] Key type ._llcrypt registered [ 7417.606530] Key type .llcrypt registered [ 7417.806569] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7419.336201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7419.432758] alg: No test for adler32 (adler32-zlib) [ 7420.666159] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7420.988446] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7422.736906] Key type lgssc registered [ 7425.059683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7436.584169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7436.608409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.040842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.473500] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7447.529170] LustreError: 272309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.545770] LustreError: 272309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7449.681316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.334065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7453.676873] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7461.862237] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7467.475328] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7474.420758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7474.791909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.765266] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7482.998284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7488.179292] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.037766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.055168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.075726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.081888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7514.089961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.092683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.118797] Lustre: Skipped 1 previous similar message [ 7517.574319] Lustre: server umount lustre-OST0000 complete [ 7524.321078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.322635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.349752] Lustre: Skipped 1 previous similar message [ 7524.382255] Lustre: Skipped 1 previous similar message [ 7528.442896] Lustre: server umount lustre-MDT0000 complete [ 7529.441318] LustreError: 272305:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.462144] LustreError: 272305:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7532.193753] LustreError: 272291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134273 with bad export cookie 13682683284026621203 [ 7532.196470] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7532.202496] LustreError: 272291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7532.587813] Lustre: server umount lustre-MDT0001 complete [ 7544.025288] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7546.846754] Key type lgssc unregistered [ 7547.160797] LNet: 276616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.174320] LNetError: 276616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.193735] LNet: Removed LNI 192.168.203.106@tcp [ 7548.065221] Key type .llcrypt unregistered [ 7548.071935] Key type ._llcrypt unregistered [ 7565.786722] Key type ._llcrypt registered [ 7565.789935] Key type .llcrypt registered [ 7565.939264] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7568.132324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.269483] alg: No test for adler32 (adler32-zlib) [ 7569.684957] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7570.280081] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7572.024177] Key type lgssc registered [ 7573.944690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7584.032874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7584.051936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.729141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.345452] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7596.021133] LustreError: 277859:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.049326] LustreError: 277859:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7601.055294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.122197] LustreError: 277861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.618321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.316393] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7615.620322] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7621.105791] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7628.392497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.764345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.845608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7635.835858] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7642.391568] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.538311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.540518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.543229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7661.547523] LustreError: Skipped 1 previous similar message [ 7661.561648] Lustre: Skipped 1 previous similar message [ 7663.212430] Lustre: server umount lustre-OST0000 complete [ 7668.192570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7668.202604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.222655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.229800] Lustre: Skipped 1 previous similar message [ 7671.780606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.781898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.121915] Lustre: server umount lustre-MDT0000 complete [ 7676.299719] LustreError: 277825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134417 with bad export cookie 14332606393173083430 [ 7676.308889] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.479297] Lustre: server umount lustre-MDT0001 complete [ 7687.461373] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7691.043397] Key type lgssc unregistered [ 7691.291977] LNet: 281562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7691.296873] LNetError: 281562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7691.308617] LNet: Removed LNI 192.168.203.106@tcp [ 7692.153955] Key type .llcrypt unregistered [ 7692.156604] Key type ._llcrypt unregistered [ 7705.978742] Key type ._llcrypt registered [ 7705.983909] Key type .llcrypt registered [ 7706.135719] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7723.398542] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 7738.549524] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7739.773474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7739.841285] alg: No test for adler32 (adler32-zlib) [ 7740.915250] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 7741.261050] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7742.994133] Key type lgssc registered [ 7743.971860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7776.763723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7776.799452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.173969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7778.213722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7778.346778] Lustre: lustre-MDT0000: new disk, initializing [ 7778.487250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.508828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7783.738309] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7794.800567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.985824] Lustre: 285118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7795.058745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7795.063748] Lustre: Skipped 1 previous similar message [ 7795.167726] Lustre: lustre-MDT0001: new disk, initializing [ 7795.308703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7795.353058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7795.364360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7800.556364] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7810.197564] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7815.729497] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7822.007452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.232602] Lustre: lustre-OST0000: new disk, initializing [ 7822.236462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7822.242768] Lustre: 287086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7822.328280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.363792] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7831.689607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7831.700628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7831.794727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7837.617189] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7843.038662] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7843.316830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7847.648905] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7848.128349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7852.512368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.513502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.522239] LustreError: Skipped 1 previous similar message [ 7852.523301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.580962] Lustre: Skipped 1 previous similar message [ 7857.635123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7857.640035] Lustre: Skipped 2 previous similar messages [ 7862.754274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7862.773463] Lustre: Skipped 1 previous similar message [ 7864.287201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.425574] Lustre: server umount lustre-OST0000 complete [ 7871.971565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.994278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.020156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7872.994537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.796859] Lustre: server umount lustre-MDT0000 complete [ 7879.042912] LustreError: 285110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134619 with bad export cookie 17665636288499156458 [ 7879.050896] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7879.065685] LustreError: 285110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7879.594731] Lustre: server umount lustre-MDT0001 complete [ 7887.948884] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:50:26 (1789134626) [ 7894.858356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7895.337590] LustreError: 289497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.354590] LustreError: 289497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7895.431930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.962056] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7900.641932] LustreError: 289498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7905.763204] LustreError: 289497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.340925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.694095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.833688] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7919.843747] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7925.282682] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7931.486515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.778906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.188311] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7945.110735] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7957.477463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7957.487977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.492057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7957.503797] Lustre: Skipped 1 previous similar message [ 7957.513018] Lustre: Skipped 3 previous similar messages [ 7961.506210] Lustre: server umount lustre-OST0000 complete [ 7967.718799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.750658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7967.760257] Lustre: Skipped 1 previous similar message [ 7971.802517] Lustre: server umount lustre-MDT0000 complete [ 7972.839392] LustreError: 289498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.860128] LustreError: 289498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7975.181845] LustreError: 290232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134716 with bad export cookie 17665636288499157340 [ 7975.193830] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.198966] LustreError: 290232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7975.675701] Lustre: server umount lustre-MDT0001 complete [ 7986.494248] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7989.116208] Key type lgssc unregistered [ 7989.508687] LNet: 293036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.518273] LNetError: 293036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.545434] LNet: Removed LNI 192.168.203.106@tcp [ 7990.423920] Key type .llcrypt unregistered [ 7990.434472] Key type ._llcrypt unregistered [ 8012.615175] Key type ._llcrypt registered [ 8012.620159] Key type .llcrypt registered [ 8012.730850] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8026.745126] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8028.114962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8028.336063] alg: No test for adler32 (adler32-zlib) [ 8029.565713] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8029.824356] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8031.618460] Key type lgssc registered [ 8033.326185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8078.507983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8078.530716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8079.828545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8079.873499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8079.983991] Lustre: lustre-MDT0000: new disk, initializing [ 8080.087644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.113475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8085.897665] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8097.188703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.329412] Lustre: 296864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8097.373297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8097.376122] Lustre: Skipped 1 previous similar message [ 8097.469176] Lustre: lustre-MDT0001: new disk, initializing [ 8097.595769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8097.624814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8097.649737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8102.296975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8113.931104] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8121.920737] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8129.580991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.878611] Lustre: lustre-OST2710: new disk, initializing [ 8129.881343] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8129.886928] Lustre: 298835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8129.958294] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8134.930675] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8140.385110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8140.398808] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8140.501669] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8144.369587] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8152.028411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.272981] Lustre: lustre-OST03e8: new disk, initializing [ 8152.281899] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8152.288827] Lustre: 299900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8152.412477] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8160.209262] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8160.956930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8160.969506] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8161.125664] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8168.036780] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.097725] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8175.317300] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8179.724203] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8179.908084] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8184.457204] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8184.910956] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8190.447945] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8190.626724] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8195.040652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.051345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.069228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.090370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.105900] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.122973] Lustre: Skipped 1 previous similar message [ 8200.477680] Lustre: server umount lustre-MDT0000 complete [ 8203.751469] LustreError: 296857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134944 with bad export cookie 16543471983207131904 [ 8203.761303] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8203.763487] LustreError: 296857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8204.073249] Lustre: server umount lustre-MDT0001 complete [ 8217.253382] Lustre: server umount lustre-OST2710 complete [ 8231.380672] Lustre: server umount lustre-OST03e8 complete [ 8254.861159] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8265.656516] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8301.020699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.324992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8301.372940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8301.460821] Lustre: lustre-MDT0000: new disk, initializing [ 8301.512081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8301.533743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8306.838341] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8317.304675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.418682] Lustre: 305150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8317.515328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8317.524204] Lustre: Skipped 1 previous similar message [ 8317.652374] Lustre: lustre-MDT0001: new disk, initializing [ 8317.744087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8317.782158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8317.791688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8322.039540] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8330.984206] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8336.428746] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8343.427572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8343.785686] Lustre: lustre-OST0000: new disk, initializing [ 8343.790271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8343.799107] Lustre: 307121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8343.906744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.199902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8349.209734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8349.307368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8350.054989] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8359.933981] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8366.076292] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8366.483460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8371.436522] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8371.694963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8374.757762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.759320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.784327] Lustre: Skipped 1 previous similar message [ 8374.799979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.809238] Lustre: Skipped 4 previous similar messages [ 8379.884904] Lustre: server umount lustre-OST0000 complete [ 8384.481525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8384.496055] LustreError: Skipped 1 previous similar message [ 8384.504093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8384.520970] Lustre: Skipped 1 previous similar message [ 8384.529821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.753722] Lustre: server umount lustre-MDT0000 complete [ 8390.115557] LustreError: 305162:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.132707] LustreError: 305162:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 8393.772447] LustreError: 305142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135134 with bad export cookie 16543471983207133584 [ 8393.775692] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.778098] LustreError: 305142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8393.998789] Lustre: server umount lustre-MDT0001 complete [ 8403.936523] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:59:02 (1789135142) [ 8405.290434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8406.661465] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:59:06 (1789135146) [ 8409.818091] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8435.343610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.587801] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8435.603613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8435.816419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.146082] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8451.296656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.514889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8451.566611] Lustre: 310979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8458.060607] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8478.964624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.306359] Lustre: Denying initial registration attempt from nid 192.168.203.106@tcp, specified as failover [ 8479.321685] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8479.342207] LustreError: 312431:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8479.350201] LustreError: 312431:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8479.357575] LustreError: 312431:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8479.483300] Lustre: server umount lustre-OST0000 complete [ 8479.489344] LustreError: 312431:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8487.407284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.413264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.414559] Lustre: Skipped 2 previous similar messages [ 8487.438714] Lustre: Skipped 2 previous similar messages [ 8491.615940] Lustre: server umount lustre-MDT0000 complete [ 8492.513026] LustreError: 310990:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.537512] LustreError: 310990:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8495.455301] LustreError: 310969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135236 with bad export cookie 16543471983207134466 [ 8495.474865] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.475879] LustreError: 310969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8495.904473] Lustre: server umount lustre-MDT0001 complete [ 8508.517284] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8511.708291] Key type lgssc unregistered [ 8512.060848] LNet: 313557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8512.072810] LNetError: 313557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.127468] LNet: Removed LNI 192.168.203.106@tcp [ 8514.042919] Key type .llcrypt unregistered [ 8514.045589] Key type ._llcrypt unregistered [ 8526.721727] Key type ._llcrypt registered [ 8526.723682] Key type .llcrypt registered [ 8526.875382] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8528.358710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8528.695727] alg: No test for adler32 (adler32-zlib) [ 8529.951978] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8530.475399] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8532.151281] Key type lgssc registered [ 8533.744266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8583.358887] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8592.203977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8592.221256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.444532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8593.470925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8593.663275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8596.849334] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8605.150600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.284565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8605.360622] Lustre: 316336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8605.777620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.108157] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8617.486776] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8623.496099] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8644.858159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.970803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8645.192108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.349200] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8661.963079] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8665.574187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.575291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8665.575727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8665.593251] Lustre: Skipped 1 previous similar message [ 8670.690476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8670.697517] Lustre: Skipped 2 previous similar messages [ 8675.810084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8675.828900] Lustre: Skipped 1 previous similar message [ 8678.879134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8679.045589] Lustre: server umount lustre-OST0000 complete [ 8686.049390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.064151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.072125] Lustre: Skipped 1 previous similar message [ 8689.707714] Lustre: server umount lustre-MDT0000 complete [ 8694.585395] LustreError: 316327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135435 with bad export cookie 12612481600671184121 [ 8694.602341] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8694.625214] LustreError: 316327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8695.531662] Lustre: server umount lustre-MDT0001 complete [ 8715.033433] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8718.058197] Key type lgssc unregistered [ 8718.399512] LNet: 319939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8718.407059] LNetError: 319939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8718.422976] LNet: Removed LNI 192.168.203.106@tcp [ 8719.397956] Key type .llcrypt unregistered [ 8719.403899] Key type ._llcrypt unregistered [ 8748.375836] Key type ._llcrypt registered [ 8748.378749] Key type .llcrypt registered [ 8748.470593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8749.959219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8750.202372] alg: No test for adler32 (adler32-zlib) [ 8751.549723] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8752.162227] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8753.929224] Key type lgssc registered [ 8755.605584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8765.112642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8765.127950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8766.461526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8771.490572] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8776.674410] LustreError: 321357:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.719381] LustreError: 321357:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8780.231936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.667586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8785.016930] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8790.996305] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.359530] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.321723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8806.724380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.233085] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8823.727113] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8831.979087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8831.987633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.003643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.554641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.556581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.563032] Lustre: Skipped 1 previous similar message [ 8835.582856] Lustre: Skipped 2 previous similar messages [ 8836.932688] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 8838.035222] Lustre: server umount lustre-MDT0000 complete [ 8840.679103] LustreError: 321357:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.694355] LustreError: 321357:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8841.153529] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135582 with bad export cookie 3089216365456276268 [ 8841.164372] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8841.170807] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8842.029087] LustreError: 321381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.052306] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 8845.792066] LustreError: 321354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.792449] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.817063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8846.831379] Lustre: Skipped 1 previous similar message [ 8847.424962] Lustre: server umount lustre-MDT0001 complete [ 8853.105115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8860.763339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.276593] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2adf19eac1585955 [ 8866.295082] Lustre: MGC192.168.203.106@tcp: Connection restored to 0@lo (at 0@lo) [ 8866.659433] LustreError: 323190:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.675735] LustreError: 323190:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8866.761374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.817609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8871.908203] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8876.975691] LustreError: 324951:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.983160] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8876.991552] LustreError: 324951:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 8880.615164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.070177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.705544] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8886.267630] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8893.196647] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.611574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8899.982914] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.742880] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8907.744785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8907.749237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.759038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8907.762794] Lustre: Skipped 1 previous similar message [ 8911.842574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8911.845082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.055337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8921.188556] Lustre: server umount lustre-OST0000 complete [ 8927.201321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.205379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8927.222726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.222812] Lustre: Skipped 1 previous similar message [ 8927.246298] Lustre: Skipped 5 previous similar messages [ 8931.661575] Lustre: server umount lustre-MDT0000 complete [ 8932.322110] LustreError: 324952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8932.335373] LustreError: 324952:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8935.192990] LustreError: 321338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135676 with bad export cookie 3089216365456283989 [ 8935.197695] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8935.202764] LustreError: 321338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8935.586105] Lustre: server umount lustre-MDT0001 complete [ 8947.770134] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8950.761579] Key type lgssc unregistered [ 8951.144991] LNet: 327629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8951.155922] LNetError: 327629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8951.178899] LNet: Removed LNI 192.168.203.106@tcp [ 8952.124932] Key type .llcrypt unregistered [ 8952.128183] Key type ._llcrypt unregistered [ 8980.939465] Key type ._llcrypt registered [ 8980.942071] Key type .llcrypt registered [ 8981.069393] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8982.427910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8982.486454] alg: No test for adler32 (adler32-zlib) [ 8983.547711] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 8983.852091] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8985.535167] Key type lgssc registered [ 8987.017564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8994.677640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8994.692945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.234370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.076833] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9006.562660] LustreError: 329047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.591740] LustreError: 329047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9009.159575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.439890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9015.379860] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9023.163974] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9029.844977] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9034.277690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.286785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9034.294424] Lustre: Skipped 1 previous similar message [ 9034.309078] Lustre: Skipped 1 previous similar message [ 9038.509899] Lustre: server umount lustre-MDT0000 complete [ 9039.337163] LustreError: 330666:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.363696] LustreError: 330666:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9042.126920] LustreError: 329027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135783 with bad export cookie 9354501207825693585 [ 9042.130054] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9042.133707] LustreError: 329027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9042.431492] Lustre: server umount lustre-MDT0001 complete [ 9055.520619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.879298] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9055.911075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9056.136250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.656284] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9068.907808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.061174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9069.143551] Lustre: 331636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9069.545992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.978250] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9080.735829] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9086.244857] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9095.856676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9096.091360] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9096.102721] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9096.108366] LustreError: 333520:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9096.112734] LustreError: 333520:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9096.116955] LustreError: 333520:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9096.162673] Lustre: server umount lustre-OST0000 complete [ 9096.167423] LustreError: 333520:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9106.348340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9106.681464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9106.937795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.839902] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9115.415524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9124.401766] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.819151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.978752] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9137.160343] Lustre: lustre-OST0001: new disk, initializing [ 9137.165919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9137.185904] Lustre: 335063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9137.345268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9145.190961] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9147.439396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9147.451240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9147.582317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9154.612423] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9158.112205] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9158.120264] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.123792] LustreError: Skipped 1 previous similar message [ 9158.128539] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9158.145415] Lustre: Skipped 1 previous similar message [ 9163.250787] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9163.257300] Lustre: Skipped 2 previous similar messages [ 9164.076265] Lustre: server umount lustre-OST0001 complete [ 9168.355806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.357127] LustreError: 333986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9168.382096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9168.403174] LustreError: 333986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9168.436647] Lustre: Skipped 1 previous similar message [ 9173.479334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.480273] LustreError: 333986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9173.485960] Lustre: Skipped 1 previous similar message [ 9173.504310] LustreError: 333986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9174.409992] Lustre: server umount lustre-OST0000 complete [ 9178.593125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.607775] Lustre: Skipped 1 previous similar message [ 9183.719472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.727448] Lustre: Skipped 3 previous similar messages [ 9184.618832] Lustre: server umount lustre-MDT0000 complete [ 9188.576939] LustreError: 331628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135929 with bad export cookie 9354501207825694145 [ 9188.600975] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.605230] LustreError: 331628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9189.156432] Lustre: server umount lustre-MDT0001 complete [ 9200.783583] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9203.799648] Key type lgssc unregistered [ 9204.129892] LNet: 337025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9204.134928] LNetError: 337025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9204.167832] LNet: Removed LNI 192.168.203.106@tcp [ 9205.284099] Key type .llcrypt unregistered [ 9205.286055] Key type ._llcrypt unregistered [ 9279.632225] Key type ._llcrypt registered [ 9279.638079] Key type .llcrypt registered [ 9279.816388] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 9295.165538] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9296.651838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9296.720272] alg: No test for adler32 (adler32-zlib) [ 9297.790517] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 9298.113547] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9299.799214] Key type lgssc registered [ 9300.857049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9333.479156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9333.500953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.719116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9334.757901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9334.901993] Lustre: lustre-MDT0000: new disk, initializing [ 9335.038074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9335.065149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9340.253556] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9351.314892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.446898] Lustre: 341959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9351.522552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9351.526136] Lustre: Skipped 1 previous similar message [ 9351.624968] Lustre: lustre-MDT0001: new disk, initializing [ 9351.712974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9351.741254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9351.747822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9356.296671] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9365.775622] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9371.685964] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9379.219034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.596683] Lustre: lustre-OST0000: new disk, initializing [ 9379.602331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9379.611585] Lustre: 343928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.685452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.554123] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9386.625985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9386.649577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9386.825582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9395.571436] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9399.777932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9400.044484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9404.140994] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9404.454325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9407.464616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9407.466719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9407.496529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.573494] Lustre: server umount lustre-OST0000 complete [ 9417.697914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.715355] Lustre: Skipped 1 previous similar message [ 9417.734618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9417.746165] Lustre: Skipped 2 previous similar messages [ 9422.426586] Lustre: server umount lustre-MDT0000 complete [ 9426.086152] LustreError: 341953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136166 with bad export cookie 6485433117326855061 [ 9426.090988] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9426.096263] LustreError: 341953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9426.423269] Lustre: server umount lustre-MDT0001 complete [ 9434.523187] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:16:13 (1789136173) [ 9441.940392] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:16:21 (1789136181) [ 9450.220047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.744583] LustreError: 346685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.771131] LustreError: 346685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9450.849586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.678736] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9456.096614] LustreError: 346686:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9460.203896] LustreError: 346685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.095963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.488880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.304790] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9476.403213] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9482.370327] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.265516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.771555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.238383] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9503.444526] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9513.530406] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9515.276912] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9516.783374] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9518.379282] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9520.031930] Lustre: Failing over lustre-MDT0000 [ 9520.271086] Lustre: server umount lustre-MDT0000 complete [ 9520.456213] LustreError: 346681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.608674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9520.617227] LustreError: Skipped 1 previous similar message [ 9520.621378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.631361] Lustre: Skipped 1 previous similar message [ 9524.704586] LustreError: 346686:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.718825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.722350] LustreError: 346686:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9534.946070] LustreError: 346685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.970426] LustreError: 346685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9538.185916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.322946] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.602482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.650712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9540.905863] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9542.923225] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9543.675357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9543.716816] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9543.770894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9552.293829] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.048755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9557.575755] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9559.533572] Lustre: Failing over lustre-MDT0000 [ 9559.773818] Lustre: server umount lustre-MDT0000 complete [ 9561.427306] LustreError: 348504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.453215] LustreError: 348504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9564.130772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.151898] Lustre: Skipped 3 previous similar messages [ 9564.164310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.970629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9579.077497] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9579.306596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.372753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9581.863169] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9584.179380] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9584.627989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9584.635617] Lustre: Skipped 2 previous similar messages [ 9584.683969] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9584.733329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9593.189554] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9594.662282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9605.094714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9605.097423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.117119] Lustre: Skipped 2 previous similar messages [ 9605.118229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9605.131455] Lustre: Skipped 2 previous similar messages [ 9607.719222] Lustre: server umount lustre-MDT0000 complete [ 9610.212789] LustreError: 347413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9610.229541] LustreError: 347413:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 9612.055678] LustreError: 348528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136352 with bad export cookie 6485433117326857770 [ 9612.078188] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9612.084558] LustreError: 348528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9612.465945] Lustre: server umount lustre-MDT0001 complete [ 9625.474170] Lustre: server umount lustre-OST0000 complete [ 9640.031744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.834866] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9654.730979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.058128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9655.067200] Lustre: Skipped 1 previous similar message [ 9659.664469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9668.146360] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9674.941654] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9682.541317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9686.011085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9689.500565] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9697.272274] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9709.027437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9709.038634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.063907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9713.807996] Lustre: server umount lustre-OST0000 complete [ 9719.787733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.798257] Lustre: Skipped 2 previous similar messages [ 9723.995461] Lustre: server umount lustre-MDT0000 complete [ 9724.899929] LustreError: 353021:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9724.919520] LustreError: 353021:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 9727.451086] LustreError: 353004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136468 with bad export cookie 6485433117326858505 [ 9727.461402] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.466847] LustreError: 353004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9727.785219] Lustre: server umount lustre-MDT0001 complete [ 9738.434407] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 9741.077259] Key type lgssc unregistered [ 9741.382945] LNet: 356561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9741.399665] LNetError: 356561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9741.422407] LNet: Removed LNI 192.168.203.106@tcp [ 9742.157483] Key type .llcrypt unregistered [ 9742.160771] Key type ._llcrypt unregistered [ 9767.760683] Key type ._llcrypt registered [ 9767.768081] Key type .llcrypt registered [ 9767.934817] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 9769.247804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9769.287074] alg: No test for adler32 (adler32-zlib) [ 9770.346195] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [ 9770.608384] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 9772.335290] Key type lgssc registered [ 9773.601222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9781.165284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9781.183883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.681513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9787.177324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9793.002169] LustreError: 358096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.049569] LustreError: 358096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9795.193086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9795.713860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9799.927130] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9806.881603] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9812.795629] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9819.931889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9820.177964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.415674] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9827.392986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9834.269124] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9846.753517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9846.765170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9846.779109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9847.781319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9847.786820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.811233] Lustre: Skipped 1 previous similar message [ 9850.444234] Lustre: server umount lustre-MDT0000 complete [ 9852.903228] LustreError: 358091:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.915441] LustreError: 358091:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9853.361549] LustreError: 358078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136594 with bad export cookie 6941335454242666071 [ 9853.379871] LustreError: 358078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9853.381406] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9853.621561] Lustre: server umount lustre-MDT0001 complete [ 9866.427893] Lustre: server umount lustre-OST0000 complete [ 9877.971885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.307178] LustreError: 361807:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9878.374915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.476660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9883.618731] LustreError: 361808:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.741592] LustreError: 361807:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.476371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.865362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9894.241431] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9901.867812] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9906.305857] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9912.118136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9912.401219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.585748] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9924.022546] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9924.725275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9940.450797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.452480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9940.466670] Lustre: Skipped 3 previous similar messages [ 9940.470107] Lustre: Skipped 2 previous similar messages [ 9945.570289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9945.574501] Lustre: Skipped 3 previous similar messages [ 9945.733957] Lustre: server umount lustre-MDT0000 complete [ 9948.969832] LustreError: 361788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136689 with bad export cookie 6941335454242667184 [ 9948.976156] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.992941] LustreError: 361788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9949.325463] Lustre: server umount lustre-MDT0001 complete [ 9962.373670] Lustre: server umount lustre-OST0000 complete [ 9977.663762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9978.115713] LustreError: 365717:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9978.194037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.082669] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 9990.728411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.851140] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10001.501220] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10006.527253] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10013.092967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.477490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10013.485891] Lustre: Skipped 1 previous similar message [10019.639469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10023.736176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10026.372554] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10031.074538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10031.081530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.104318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10034.147399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10039.271385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10039.280087] Lustre: Skipped 2 previous similar messages [10044.895785] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10045.109738] Lustre: server umount lustre-OST0000 complete [10049.511915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.536722] Lustre: Skipped 1 previous similar message [10054.612640] Lustre: server umount lustre-MDT0000 complete [10054.629572] LustreError: 365713:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.644163] LustreError: 365713:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10057.758744] LustreError: 365697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136798 with bad export cookie 6941335454242668206 [10057.780762] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10058.069527] Lustre: server umount lustre-MDT0001 complete [10066.967536] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10069.531375] Key type lgssc unregistered [10069.798841] LNet: 369256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10069.803557] LNetError: 369256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10069.819137] LNet: Removed LNI 192.168.203.106@tcp [10070.561093] Key type .llcrypt unregistered [10070.563800] Key type ._llcrypt unregistered [10092.420237] Key type ._llcrypt registered [10092.421489] Key type .llcrypt registered [10092.529239] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10093.499541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10093.549692] alg: No test for adler32 (adler32-zlib) [10094.564728] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [10094.776275] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10096.415388] Key type lgssc registered [10097.237498] Lustre: Echo OBD driver; http://www.lustre.org/ [10102.999530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10103.014983] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.022971] LustreError: 370697:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10103.040936] LustreError: 370697:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10103.045953] LustreError: 370697:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10103.052439] LustreError: 370697:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10103.060735] LustreError: 370697:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10108.167818] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10108.173353] LustreError: 371094:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10108.185859] LustreError: 371094:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10108.193252] LustreError: 371094:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10108.198279] LustreError: 371094:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10108.203706] LustreError: 371094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10122.673758] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10124.843153] Key type lgssc unregistered [10125.093265] LNet: 372100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10125.104473] LNetError: 372100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10125.118948] LNet: Removed LNI 192.168.203.106@tcp [10125.716482] Key type .llcrypt unregistered [10125.722857] Key type ._llcrypt unregistered [10139.137626] Key type ._llcrypt registered [10139.139524] Key type .llcrypt registered [10139.233636] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10150.268320] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10150.979497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10151.005590] alg: No test for adler32 (adler32-zlib) [10152.126829] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [10152.327808] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10153.951211] Key type lgssc registered [10155.047647] Lustre: Echo OBD driver; http://www.lustre.org/ [10182.795263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10182.847827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.057070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10184.116699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10184.213094] Lustre: lustre-MDT0000: new disk, initializing [10184.284313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10184.310792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10187.710391] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10197.062052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.175456] Lustre: 375236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10197.225378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10197.229104] Lustre: Skipped 1 previous similar message [10197.280169] Lustre: lustre-MDT0001: new disk, initializing [10197.363736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.392361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10197.406724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10201.480756] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10209.684394] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10214.409527] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10220.136428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.356610] Lustre: lustre-OST0000: new disk, initializing [10220.359427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10220.363905] Lustre: 377202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10220.421099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10225.207534] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10227.774039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10227.786268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10227.862342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10232.877530] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10236.753592] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10236.921154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10240.370733] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10240.514189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10243.560923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.561356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10243.562387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10243.573034] Lustre: Skipped 1 previous similar message [10243.576076] LustreError: Skipped 1 previous similar message [10248.675071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10248.685893] Lustre: Skipped 2 previous similar messages [10253.795970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10255.842619] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10256.045669] Lustre: server umount lustre-OST0000 complete [10264.034295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.034581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10264.052824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.067651] Lustre: Skipped 1 previous similar message [10269.155455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10269.158088] Lustre: Skipped 1 previous similar message [10274.271124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10274.424424] Lustre: server umount lustre-MDT0000 complete [10277.346781] LustreError: 377201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137018 with bad export cookie 15044513398163243471 [10277.352276] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10277.354576] LustreError: 377201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10277.605615] Lustre: server umount lustre-MDT0001 complete [10284.441798] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10285.758276] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:30:25 (1789137025) [10291.636345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.983631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.868810] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10297.317322] LustreError: 379665:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.339743] LustreError: 379665:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10301.414907] LustreError: 379664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.181067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.458357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10307.151481] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10312.963972] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10317.171890] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10322.883512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.128276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.461453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10338.121078] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10344.428224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.634963] Lustre: lustre-OST0001: new disk, initializing [10344.653640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10344.661748] Lustre: 382501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10344.790535] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10350.147792] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10352.239864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10352.247959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10352.333407] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10358.091655] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10362.849950] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10362.866481] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.873630] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10362.891840] Lustre: Skipped 2 previous similar messages [10362.896027] Lustre: Skipped 2 previous similar messages [10367.860284] Lustre: server umount lustre-OST0001 complete [10367.974255] LustreError: 381781:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.986834] LustreError: 381781:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10373.092587] LustreError: 381484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.117211] LustreError: 381484:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10378.212133] LustreError: 381781:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.238556] LustreError: 381781:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10384.865076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10384.868228] LustreError: Skipped 1 previous similar message [10384.876520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10384.895171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10384.904880] Lustre: Skipped 1 previous similar message [10388.459172] LustreError: 381487:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.488562] LustreError: 381487:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10390.280540] Lustre: server umount lustre-OST0000 complete [10393.568872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.593385] Lustre: Skipped 2 previous similar messages [10401.618365] Lustre: server umount lustre-MDT0000 complete [10405.202353] LustreError: 380398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137146 with bad export cookie 15044513398163244353 [10405.217091] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10405.218585] LustreError: 380398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10411.680771] Lustre: server umount lustre-MDT0001 complete [10420.981462] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10423.159887] Key type lgssc unregistered [10423.454652] LNet: 384489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10423.461113] LNetError: 384489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10423.479333] LNet: Removed LNI 192.168.203.106@tcp [10424.312134] Key type .llcrypt unregistered [10424.319229] Key type ._llcrypt unregistered [10464.994447] Key type ._llcrypt registered [10464.997667] Key type .llcrypt registered [10465.077907] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10465.698978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10465.718244] alg: No test for adler32 (adler32-zlib) [10466.787761] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [10467.002560] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10468.696579] Key type lgssc registered [10469.653884] Lustre: Echo OBD driver; http://www.lustre.org/ [10477.379849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10477.402272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.697622] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10478.719326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10478.846191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10482.930717] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10490.143774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.215635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10490.234291] Lustre: 387284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10490.469214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10494.444157] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10500.263412] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10504.831964] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10506.722639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.739200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.754230] Lustre: Skipped 1 previous similar message [10511.843485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10511.851928] Lustre: Skipped 1 previous similar message [10512.902693] Lustre: server umount lustre-MDT0000 complete [10515.921571] LustreError: 387276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137256 with bad export cookie 13415870242191814935 [10515.933446] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10515.933587] LustreError: 387276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10516.179807] Lustre: server umount lustre-MDT0001 complete [10523.901502] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:34:23 (1789137263) [10529.411204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.896915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10534.214316] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10535.397661] LustreError: 389795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.416394] LustreError: 389795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10540.512629] LustreError: 389794:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.262297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.495966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10545.818565] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10552.077417] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10557.218970] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10563.114868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.259727] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10563.439423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10569.322854] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10576.105703] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10582.242910] Lustre: Permanently deactivating lustre-OST0000 [10582.259741] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10590.215590] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10593.306913] LustreError: 392828:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10593.317481] LustreError: 392828:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10593.323406] LustreError: 392828:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10596.381519] LustreError: 392876:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10596.395311] LustreError: 392876:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10601.425143] Lustre: server umount lustre-OST0000 complete [10604.350860] LustreError: 393126:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10604.357095] LustreError: 393126:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10608.100127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10608.102187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10608.112124] Lustre: Skipped 1 previous similar message [10608.113495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10608.145669] Lustre: Skipped 1 previous similar message [10613.215975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10613.219098] Lustre: Skipped 1 previous similar message [10616.372927] Lustre: server umount lustre-MDT0000 complete [10618.342366] LustreError: 389795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.384856] LustreError: 389795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10619.850969] LustreError: 389776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137360 with bad export cookie 13415870242191815509 [10619.866254] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10619.868662] LustreError: 389776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10630.379459] Lustre: server umount lustre-MDT0001 complete [10637.448334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.419350] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10643.758601] LustreError: 394285:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10648.812995] Lustre: 394623:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.106@tcp [10650.434965] Lustre: server umount MGS complete [10659.581518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.952195] LustreError: 395347:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.963881] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10659.969450] Lustre: Skipped 1 previous similar message [10660.002788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.901667] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10665.442638] LustreError: 395348:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.059534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.367034] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10676.122276] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10682.608938] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10687.706429] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10689.277274] Lustre: Permanently reactivating lustre-OST0000 [10689.295503] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10689.303258] Lustre: Skipped 2 previous similar messages [10718.178533] LustreError: 397176:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10725.721944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10725.987919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10725.995110] Lustre: Skipped 1 previous similar message [10731.040876] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10736.727235] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10742.752911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10742.770327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.790227] Lustre: Skipped 1 previous similar message [10742.796396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10743.779172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10746.602863] Lustre: server umount lustre-OST0000 complete [10753.505830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10753.511599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10753.531295] Lustre: Skipped 1 previous similar message [10753.536950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10753.543559] Lustre: Skipped 2 previous similar messages [10756.096192] Lustre: server umount lustre-MDT0000 complete [10758.979403] LustreError: 395329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137499 with bad export cookie 13415870242191816566 [10758.984205] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.988878] LustreError: 395329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10759.345350] Lustre: server umount lustre-MDT0001 complete [10768.162672] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [10771.022648] Key type lgssc unregistered [10771.301135] LNet: 399855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10771.310132] LNetError: 399855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10771.332355] LNet: Removed LNI 192.168.203.106@tcp [10772.302296] Key type .llcrypt unregistered [10772.304204] Key type ._llcrypt unregistered [10789.809537] Key type ._llcrypt registered [10789.811847] Key type .llcrypt registered [10789.912687] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [10804.182482] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [10805.378517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10805.425482] alg: No test for adler32 (adler32-zlib) [10806.535969] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [10806.764272] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [10808.463631] Key type lgssc registered [10809.572495] Lustre: Echo OBD driver; http://www.lustre.org/ [10840.312900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10840.339070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.540943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10841.566077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10841.649979] Lustre: lustre-MDT0000: new disk, initializing [10841.737873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10841.758192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10846.286386] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10855.351259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.430618] Lustre: 403011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10855.506825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10855.515489] Lustre: Skipped 1 previous similar message [10855.584734] Lustre: lustre-MDT0001: new disk, initializing [10855.705315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10855.751345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10855.761490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10859.168452] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10866.978252] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10871.793595] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10877.326911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10877.594363] Lustre: lustre-OST0000: new disk, initializing [10877.598342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10877.605702] Lustre: 404979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10877.709287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.636951] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10885.185341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10885.195545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10885.298610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10890.999255] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10895.336484] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10895.528838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10899.424426] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10899.583885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10906.082694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10906.087592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.089273] LustreError: Skipped 1 previous similar message [10906.090230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10906.111532] Lustre: Skipped 1 previous similar message [10906.138986] Lustre: Skipped 1 previous similar message [10907.470661] Lustre: server umount lustre-OST0000 complete [10911.201984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.220980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10916.336457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10916.340831] Lustre: Skipped 2 previous similar messages [10916.820417] Lustre: server umount lustre-MDT0000 complete [10920.977150] LustreError: 403003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137661 with bad export cookie 14939900510811153357 [10920.982267] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.993160] LustreError: 403003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10921.798872] Lustre: server umount lustre-MDT0001 complete [10930.184322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10931.679853] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:41:10 (1789137670) [10939.104964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.671765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10943.313807] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10944.995353] LustreError: 407440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.009502] LustreError: 407440:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10950.115151] LustreError: 407459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.898406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10951.288880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10955.490086] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10962.380767] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10968.146445] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10974.386354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.720785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10980.506586] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [10986.801150] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10988.538986] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10991.815538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10997.220241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10997.236216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10997.254207] Lustre: Skipped 1 previous similar message [10997.266556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10998.689599] Lustre: server umount lustre-OST0000 complete [11002.348597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11002.355191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.376614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.436305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.552333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11011.567787] Lustre: Skipped 3 previous similar messages [11012.675516] Lustre: server umount lustre-MDT0000 complete [11016.140323] LustreError: 408176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137757 with bad export cookie 14939900510811154239 [11016.145866] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11016.153908] LustreError: 408176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11016.651646] Lustre: server umount lustre-MDT0001 complete [11026.076587] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11029.360850] Key type lgssc unregistered [11029.589234] LNet: 411169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11029.602468] LNetError: 411169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11029.629159] LNet: Removed LNI 192.168.203.106@tcp [11030.526292] Key type .llcrypt unregistered [11030.530877] Key type ._llcrypt unregistered [11053.892964] Key type ._llcrypt registered [11053.895091] Key type .llcrypt registered [11053.996031] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11054.318796] Key type .llcrypt unregistered [11054.322203] Key type ._llcrypt unregistered [11071.662184] Key type ._llcrypt registered [11071.663864] Key type .llcrypt registered [11071.790279] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11072.708566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11072.736983] alg: No test for adler32 (adler32-zlib) [11073.864954] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11074.076605] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11075.727170] Key type lgssc registered [11076.991738] Lustre: Echo OBD driver; http://www.lustre.org/ [11084.301528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11084.319982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.854529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11090.525785] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11096.037064] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.050172] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11098.745204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.029909] LustreError: 413386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.045689] LustreError: 413386:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11099.049620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11104.236519] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.268548] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11107.114960] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11109.345441] LustreError: 413386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.377682] LustreError: 413386:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11110.389476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11115.131934] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11115.505350] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.535403] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11122.268517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.550471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11126.147932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11132.900898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11132.901705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.906738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.925439] Lustre: Skipped 1 previous similar message [11138.020468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11138.026239] Lustre: Skipped 2 previous similar messages [11138.880215] Lustre: server umount lustre-OST0000 complete [11142.110396] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11142.122648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.146756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11142.349584] Lustre: server umount lustre-MDT0000 complete [11143.137328] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11143.141209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.155632] LustreError: 413387:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11145.452223] LustreError: 413366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137886 with bad export cookie 16137116261840340433 [11145.459770] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11145.475026] LustreError: 413366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11145.911579] Lustre: server umount lustre-MDT0001 complete [11155.793404] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11158.019595] Key type lgssc unregistered [11158.341436] LNet: 416536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11158.346688] LNetError: 416536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11158.358876] LNet: Removed LNI 192.168.203.106@tcp [11159.221511] Key type .llcrypt unregistered [11159.223707] Key type ._llcrypt unregistered [11181.516894] Key type ._llcrypt registered [11181.518877] Key type .llcrypt registered [11181.594656] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11182.289295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11182.400569] alg: No test for adler32 (adler32-zlib) [11183.397983] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11183.595505] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11185.207408] Key type lgssc registered [11186.190714] Lustre: Echo OBD driver; http://www.lustre.org/ [11193.711838] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11193.748972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.312164] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11214.431822] LustreError: 417925:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [11241.119716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11246.618981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11253.887327] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11260.202709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.741398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11260.828861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11264.714436] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11266.038105] LustreError: 417950:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11266.068051] LustreError: 417950:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11271.145553] LustreError: 417951:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.423323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11273.893960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11278.567831] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11284.794380] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11290.017728] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11297.772816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.779635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11297.789469] Lustre: Skipped 1 previous similar message [11297.807535] Lustre: Skipped 1 previous similar message [11299.092899] Lustre: server umount lustre-OST0000 complete [11302.880649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.901846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11302.906867] Lustre: Skipped 1 previous similar message [11308.002706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11308.017566] Lustre: Skipped 1 previous similar message [11308.899122] Lustre: server umount lustre-MDT0000 complete [11312.644953] LustreError: 417933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138053 with bad export cookie 3779235801931213523 [11312.658569] LustreError: 417933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11312.662973] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11313.095028] Lustre: server umount lustre-MDT0001 complete [11323.389781] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11326.142645] Key type lgssc unregistered [11326.496264] LNet: 421498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11326.505506] LNetError: 421498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11326.518798] LNet: Removed LNI 192.168.203.106@tcp [11327.419069] Key type .llcrypt unregistered [11327.426388] Key type ._llcrypt unregistered [11348.777682] Key type ._llcrypt registered [11348.779726] Key type .llcrypt registered [11348.890456] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11349.713214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.810953] alg: No test for adler32 (adler32-zlib) [11351.012184] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11351.244575] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11352.951395] Key type lgssc registered [11354.004957] Lustre: Echo OBD driver; http://www.lustre.org/ [11361.385613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11361.424271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.852176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11366.598670] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11373.024255] LustreError: 422893:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.045578] LustreError: 422893:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11373.774796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11374.141736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11378.189785] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11385.091803] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11389.838938] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11396.546127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.914697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11405.259958] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11407.161615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11415.037722] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11422.701903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.712374] LustreError: 425149:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.743581] Lustre: Skipped 2 previous similar messages [11422.782972] LustreError: 425149:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11422.992383] Lustre: server umount lustre-MDT0000 complete [11425.773861] LustreError: 422874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1789138166 with bad export cookie 416385662875769188 [11427.378140] LustreError: 422876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138168 with bad export cookie 416385662875768292 [11427.385367] LustreError: 422876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11427.389144] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11427.810488] LustreError: 425149:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.812201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11427.831836] LustreError: 425149:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11427.851482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11432.930583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11432.930779] LustreError: 422894:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.947833] LustreError: 422894:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11433.572475] Lustre: server umount lustre-OST0000 complete [11438.052191] LustreError: 423622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.107143] LustreError: 423622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11440.831705] Lustre: server umount lustre-MDT0001 complete [11452.340389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11455.448035] Key type lgssc unregistered [11455.744928] LNet: 426581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11455.754309] LNetError: 426581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11455.772445] LNet: Removed LNI 192.168.203.106@tcp [11456.607402] Key type .llcrypt unregistered [11456.610611] Key type ._llcrypt unregistered [11481.281412] Key type ._llcrypt registered [11481.283356] Key type .llcrypt registered [11481.375071] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11482.717482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11482.786446] alg: No test for adler32 (adler32-zlib) [11483.867647] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11484.146531] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11485.849152] Key type lgssc registered [11486.982720] Lustre: Echo OBD driver; http://www.lustre.org/ [11494.089630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11494.123146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.732814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.235717] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11505.124723] LustreError: 428001:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11505.154912] LustreError: 428001:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11508.297582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.663791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11513.153976] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11520.831489] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11526.427956] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11532.176081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11532.658267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11539.727863] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11540.932625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11548.761507] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.805610] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11554.823301] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.849265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11556.321045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11556.325392] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.006899] Lustre: server umount lustre-MDT0001 complete [11561.441441] LustreError: 428001:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.461508] LustreError: 428001:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11566.560580] LustreError: 428002:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.565334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11566.573714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11566.574629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.574634] Lustre: Skipped 2 previous similar messages [11571.058031] Lustre: server umount lustre-OST0000 complete [11571.685596] LustreError: 428001:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.960955] Lustre: server umount lustre-MDT0000 complete [11590.206453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11593.509138] Key type lgssc unregistered [11593.866647] LNet: 431687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11593.880445] LNetError: 431687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11593.910850] LNet: Removed LNI 192.168.203.106@tcp [11595.104107] Key type .llcrypt unregistered [11595.106141] Key type ._llcrypt unregistered [11621.267487] Key type ._llcrypt registered [11621.270759] Key type .llcrypt registered [11621.448629] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [11636.635944] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11637.656421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11637.791244] alg: No test for adler32 (adler32-zlib) [11638.896728] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11639.144864] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11640.799270] Key type lgssc registered [11641.824262] Lustre: Echo OBD driver; http://www.lustre.org/ [11677.897826] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11692.872919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11692.911056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.147648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11694.195523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11694.269299] Lustre: lustre-MDT0000: new disk, initializing [11694.404220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11694.435704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11698.343256] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11711.008859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11711.435414] Lustre: lustre-OST0000: new disk, initializing [11711.440254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11711.445606] Lustre: Skipped 1 previous similar message [11711.461793] Lustre: 436721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11711.584568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11717.772131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11717.781089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11717.828137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11719.400858] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11729.432829] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11744.721922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.936219] Lustre: 435702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11745.071157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11745.250604] Lustre: lustre-MDT0001: new disk, initializing [11745.438322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11745.486386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11745.502744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11751.044577] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11759.519281] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11759.720731] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11765.025724] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11765.209562] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11771.873681] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11771.877457] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11771.892043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11771.906580] Lustre: Skipped 1 previous similar message [11771.927052] Lustre: Skipped 1 previous similar message [11777.004112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11777.019240] Lustre: Skipped 2 previous similar messages [11780.064882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11780.077191] Lustre: Skipped 2 previous similar messages [11782.444632] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [11784.159272] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11784.521386] Lustre: server umount lustre-MDT0001 complete [11790.306417] LustreError: 435716:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.344322] LustreError: 435716:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11795.427710] LustreError: 435711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11795.466884] LustreError: 435711:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11800.571812] LustreError: 435712:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.590128] LustreError: 435712:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11804.640583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11804.654920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.678572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.664969] LustreError: 438049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.968778] Lustre: server umount lustre-OST0000 complete [11810.790062] LustreError: 435715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11819.102204] Lustre: server umount lustre-MDT0000 complete [11834.168552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11837.571361] Key type lgssc unregistered [11837.973575] LNet: 440477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11837.989600] LNetError: 440477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11838.013150] LNet: Removed LNI 192.168.203.106@tcp [11839.163281] Key type .llcrypt unregistered [11839.166191] Key type ._llcrypt unregistered [11878.196986] Key type ._llcrypt registered [11878.199457] Key type .llcrypt registered [11878.316034] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [11878.633458] Key type .llcrypt unregistered [11878.636762] Key type ._llcrypt unregistered [11894.710968] Key type ._llcrypt registered [11894.713250] Key type .llcrypt registered [11894.830167] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [11896.174928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11896.213708] alg: No test for adler32 (adler32-zlib) [11897.292691] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [11897.527089] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [11899.183171] Key type lgssc registered [11900.142831] Lustre: Echo OBD driver; http://www.lustre.org/ [11925.774239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11925.818701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.124515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11927.161560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11927.254324] Lustre: lustre-MDT0000: new disk, initializing [11927.350385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11927.363175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11931.048356] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11941.583464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.655228] Lustre: 443859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11941.693222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11941.696033] Lustre: Skipped 1 previous similar message [11941.758368] Lustre: lustre-MDT0001: new disk, initializing [11941.826318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11941.852944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11941.865138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11947.472603] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11958.623419] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11964.756680] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11972.312203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11972.571343] Lustre: lustre-OST0000: new disk, initializing [11972.576239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11972.586153] Lustre: 445830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11972.687935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11978.579222] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [11980.896274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11980.917847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.057428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11988.445567] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11996.641941] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.642992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11996.647153] Lustre: Skipped 2 previous similar messages [11996.651578] Lustre: Skipped 1 previous similar message [12000.833871] Lustre: server umount lustre-MDT0000 complete [12004.892319] LustreError: 443852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138745 with bad export cookie 2984046995561207921 [12004.899885] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12004.916381] LustreError: 443852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12005.427747] Lustre: server umount lustre-MDT0001 complete [12019.172832] Lustre: server umount lustre-OST0000 complete [12030.078955] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:59:29 (1789138769) [12037.335940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12037.816343] LustreError: 447857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.846226] LustreError: 447857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12038.036328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12042.609083] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12043.236155] LustreError: 447858:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.342106] LustreError: 447857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12052.182989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12052.453232] LustreError: 447858:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12052.866641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12057.893620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12065.772376] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12071.252657] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12091.394438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12091.668617] Lustre: Found index 0 for lustre-OST0000, updating log [12091.835321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12098.353416] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12106.279993] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12108.218431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12118.502465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12118.511274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12118.532161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12118.549624] Lustre: Skipped 2 previous similar messages [12119.587585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12119.598072] Lustre: Skipped 1 previous similar message [12123.584162] Lustre: server umount lustre-OST0000 complete [12140.000226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12140.006480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.033672] Lustre: Skipped 1 previous similar message [12140.045860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12145.121909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12145.140889] Lustre: Skipped 2 previous similar messages [12145.839567] Lustre: server umount lustre-MDT0000 complete [12146.214116] LustreError: 447858:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.249820] LustreError: 447858:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12147.487130] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [12149.725287] LustreError: 449700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138890 with bad export cookie 2984046995561209314 [12149.731609] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12149.740469] LustreError: 449700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12150.129653] Lustre: server umount lustre-MDT0001 complete [12159.022869] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:01:38 (1789138898) [12778.998044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.667527] LustreError: 454282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12779.763626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.122986] LustreError: 454283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.452627] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12789.224321] LustreError: 454282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.302860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.360811] LustreError: 454283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.379567] LustreError: 454283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12795.804732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12800.885785] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12809.081627] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12814.151193] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12833.093480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12833.436747] Lustre: Found index 0 for lustre-OST0000, updating log [12833.693111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12838.769563] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [12841.314541] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [12848.689943] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12850.649261] Lustre: server umount lustre-OST0000 complete [12855.271812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12855.282439] Lustre: Skipped 1 previous similar message [12855.290058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12855.294233] Lustre: Skipped 2 previous similar messages [12856.357460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.364543] Lustre: Skipped 1 previous similar message [12858.402240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12858.414210] Lustre: Skipped 3 previous similar messages [12860.595444] Lustre: server umount lustre-MDT0000 complete [12861.474114] LustreError: 454283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.487708] LustreError: 454283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12864.802203] LustreError: 454262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139605 with bad export cookie 2984046995561210399 [12864.807731] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12864.815461] LustreError: 454262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12865.473845] Lustre: server umount lustre-MDT0001 complete [12897.433641] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [12898.207696] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [12907.257702] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [12941.536721] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:14:40 (1789139680) [12960.345888] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [12963.596122] Key type lgssc unregistered [12963.877693] LNet: 460893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12963.885486] LNetError: 460893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13087.782663] LNet: Removed LNI 192.168.203.106@tcp [13088.592764] Key type .llcrypt unregistered [13088.595403] Key type ._llcrypt unregistered [13103.211735] Key type ._llcrypt registered [13103.217256] Key type .llcrypt registered [13103.323850] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13104.160524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13104.258694] alg: No test for adler32 (adler32-zlib) [13105.421321] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [13105.627639] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [13107.255166] Key type lgssc registered [13108.568857] Lustre: Echo OBD driver; http://www.lustre.org/ [13112.611463] LNet: Added LNI 192.168.203.106@tcp100 [8/256/0/180] [13113.306996] LNet: Added LNI 192.168.203.106@tcp101 [8/256/0/180] [13114.062724] LNet: Added LNI 192.168.203.106@tcp102 [8/256/0/180] [13114.818426] LNet: Added LNI 192.168.203.106@tcp103 [8/256/0/180] [13115.553858] LNet: Added LNI 192.168.203.106@tcp104 [8/256/0/180] [13116.246437] LNet: Added LNI 192.168.203.106@tcp105 [8/256/0/180] [13117.030551] LNet: Added LNI 192.168.203.106@tcp106 [8/256/0/180] [13117.727875] LNet: Added LNI 192.168.203.106@tcp107 [8/256/0/180] [13118.376936] LNet: Added LNI 192.168.203.106@tcp108 [8/256/0/180] [13119.009856] LNet: Added LNI 192.168.203.106@tcp109 [8/256/0/180] [13119.610679] LNet: Added LNI 192.168.203.106@tcp110 [8/256/0/180] [13120.268498] LNet: Added LNI 192.168.203.106@tcp111 [8/256/0/180] [13120.950176] LNet: Added LNI 192.168.203.106@tcp112 [8/256/0/180] [13121.567053] LNet: Added LNI 192.168.203.106@tcp113 [8/256/0/180] [13122.200491] LNet: Added LNI 192.168.203.106@tcp114 [8/256/0/180] [13122.868549] LNet: Added LNI 192.168.203.106@tcp115 [8/256/0/180] [13123.562875] LNet: Added LNI 192.168.203.106@tcp116 [8/256/0/180] [13124.172440] LNet: Added LNI 192.168.203.106@tcp117 [8/256/0/180] [13124.923966] LNet: Added LNI 192.168.203.106@tcp118 [8/256/0/180] [13125.600387] LNet: Added LNI 192.168.203.106@tcp119 [8/256/0/180] [13126.370902] LNet: Added LNI 192.168.203.106@tcp120 [8/256/0/180] [13127.112244] LNet: Added LNI 192.168.203.106@tcp121 [8/256/0/180] [13127.771994] LNet: Added LNI 192.168.203.106@tcp122 [8/256/0/180] [13128.462883] LNet: Added LNI 192.168.203.106@tcp123 [8/256/0/180] [13129.187549] LNet: Added LNI 192.168.203.106@tcp124 [8/256/0/180] [13129.972590] LNet: Added LNI 192.168.203.106@tcp125 [8/256/0/180] [13130.726491] LNet: Added LNI 192.168.203.106@tcp126 [8/256/0/180] [13131.414547] LNet: Added LNI 192.168.203.106@tcp127 [8/256/0/180] [13132.193471] LNet: Added LNI 192.168.203.106@tcp128 [8/256/0/180] [13132.892963] LNet: Added LNI 192.168.203.106@tcp129 [8/256/0/180] [13133.664139] LNet: Added LNI 192.168.203.106@tcp130 [8/256/0/180] [13134.446444] LNet: Added LNI 192.168.203.106@tcp131 [8/256/0/180] [13135.150690] LNet: Added LNI 192.168.203.106@tcp132 [8/256/0/180] [13135.890534] LNet: Added LNI 192.168.203.106@tcp133 [8/256/0/180] [13136.638528] LNet: Added LNI 192.168.203.106@tcp134 [8/256/0/180] [13137.467908] LNet: Added LNI 192.168.203.106@tcp135 [8/256/0/180] [13143.111322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13143.152603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13144.296849] Lustre: 463917:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [13144.348095] LNetError: 461377:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13144.361132] LNetError: Protocol error connecting to 192.168.203.106@tcp116 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13144.382991] LNetError: Skipped 5 previous similar messages [13144.487701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13144.519832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13144.543131] LNet: 4 local NIs in recovery (showing 4): 192.168.203.106@tcp127, 192.168.203.106@tcp128, 192.168.203.106@tcp126, 192.168.203.106@tcp129 [13144.596711] Lustre: lustre-MDT0000: new disk, initializing [13144.681549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13144.711175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13145.375785] LNetError: 463950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13145.391720] LNetError: 463950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [13145.396933] LNetError: Protocol error connecting to 192.168.203.106@tcp101 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13145.413061] LNetError: Skipped 1 previous similar message [13147.616059] LNetError: 461375:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13147.628375] LNetError: 461375:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13147.641783] LNetError: Protocol error connecting to 192.168.203.106@tcp128 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13147.649743] LNetError: Skipped 5 previous similar messages [13148.896856] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13150.691121] LNetError: 461376:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13150.709831] LNetError: Protocol error connecting to 192.168.203.106@tcp127 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13158.122605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13158.292856] Lustre: 463935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13158.413242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13158.417830] Lustre: Skipped 1 previous similar message [13158.490318] Lustre: lustre-MDT0001: new disk, initializing [13158.523875] Lustre: 464737:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [13158.530519] Lustre: 464737:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13158.645452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13158.713092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13158.729950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13158.907112] LNetError: 463950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13158.924937] LNetError: 463950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13158.944065] LNetError: Protocol error connecting to 192.168.203.106@tcp127 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13158.959643] LNetError: Skipped 1 previous similar message [13163.061269] Lustre: 464947:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [13163.077301] Lustre: 464947:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13163.640775] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13172.967761] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13174.245290] LNetError: 461378:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13174.256359] LNetError: 461378:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13174.262735] LNetError: Protocol error connecting to 192.168.203.106@tcp127 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13174.276113] LNetError: Skipped 1 previous similar message [13177.966684] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13183.897573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13184.138947] Lustre: lustre-OST0000: new disk, initializing [13184.143504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13184.150716] Lustre: 465907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13184.230329] Lustre: 465912:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [13184.238536] Lustre: 465912:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13184.283592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13189.832454] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13190.174754] Lustre: 466285:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [13190.181241] Lustre: 466285:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13190.248464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13190.254502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13190.446751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13190.628063] LNetError: 463940:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13190.637349] LNetError: 463940:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13190.642794] LNetError: Protocol error connecting to 192.168.203.106@tcp110 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13190.653084] LNetError: Skipped 8 previous similar messages [13198.897296] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13204.964195] LNet: 2 local NIs in recovery (showing 2): 192.168.203.106@tcp128, 192.168.203.106@tcp127 [13205.989854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13205.991155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13205.997163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13206.001287] LustreError: Skipped 1 previous similar message [13206.009983] Lustre: Skipped 1 previous similar message [13211.106580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13211.117409] Lustre: Skipped 2 previous similar messages [13211.630341] Lustre: server umount lustre-OST0000 complete [13215.200357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13215.215452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13215.230485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13216.225569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13221.340268] Lustre: server umount lustre-MDT0000 complete [13223.393410] LNetError: 463945:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [13223.399751] LNetError: 463945:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [13223.416389] LNetError: Protocol error connecting to 192.168.203.106@tcp122 at host 192.168.203.106:988: is it running a compatible version of Lustre? [13223.435863] LNetError: Skipped 17 previous similar messages [13224.635382] LustreError: 463926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139965 with bad export cookie 11457584472444971315 [13224.641220] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13224.645651] LustreError: 463926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13224.973412] Lustre: server umount lustre-MDT0001 complete [13235.477844] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [13238.421318] Key type lgssc unregistered [13238.804843] LNet: 467827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13238.817770] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13238.834309] LNet: Removed LNI 192.168.203.106@tcp [13239.903182] LNet: Removed LNI 192.168.203.106@tcp100 [13239.906250] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13239.912986] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13239.921163] LNet: Removed LNI 192.168.203.106@tcp101 [13240.927136] LNet: Removed LNI 192.168.203.106@tcp102 [13240.929316] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13240.943692] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13240.961434] LNet: Removed LNI 192.168.203.106@tcp103 [13242.016577] LNet: Removed LNI 192.168.203.106@tcp104 [13243.039157] LNet: Removed LNI 192.168.203.106@tcp105 [13243.043219] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13243.050116] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13243.055765] LNet: Removed LNI 192.168.203.106@tcp106 [13244.065977] LNet: Removed LNI 192.168.203.106@tcp107 [13245.151164] LNet: Removed LNI 192.168.203.106@tcp108 [13246.177443] LNet: Removed LNI 192.168.203.106@tcp109 [13247.199298] LNet: Removed LNI 192.168.203.106@tcp110 [13247.206436] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13247.215319] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13247.219559] LNet: Removed LNI 192.168.203.106@tcp111 [13248.290654] LNet: Removed LNI 192.168.203.106@tcp112 [13249.311177] LNet: Removed LNI 192.168.203.106@tcp113 [13250.342434] LNet: Removed LNI 192.168.203.106@tcp114 [13251.359127] LNet: Removed LNI 192.168.203.106@tcp115 [13252.385670] LNet: Removed LNI 192.168.203.106@tcp116 [13253.407197] LNet: Removed LNI 192.168.203.106@tcp117 [13254.431684] LNet: Removed LNI 192.168.203.106@tcp118 [13255.455459] LNet: Removed LNI 192.168.203.106@tcp119 [13255.458369] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13255.462493] LNetError: 467827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13255.466936] LNet: Removed LNI 192.168.203.106@tcp120 [13256.479145] LNet: Removed LNI 192.168.203.106@tcp121 [13257.503351] LNet: Removed LNI 192.168.203.106@tcp122 [13258.527970] LNet: Removed LNI 192.168.203.106@tcp123 [13259.551161] LNet: Removed LNI 192.168.203.106@tcp124 [13260.576076] LNet: Removed LNI 192.168.203.106@tcp125 [13261.601286] LNet: Removed LNI 192.168.203.106@tcp126 [13261.603758] LNet: Removed LNI 192.168.203.106@tcp127 [13261.608288] LNet: Removed LNI 192.168.203.106@tcp128 [13262.623497] LNet: Removed LNI 192.168.203.106@tcp129 [13262.627126] LNet: Removed LNI 192.168.203.106@tcp130 [13262.633783] LNet: Removed LNI 192.168.203.106@tcp131 [13262.638197] LNet: Removed LNI 192.168.203.106@tcp132 [13262.642707] LNet: Removed LNI 192.168.203.106@tcp133 [13262.645268] LNet: Removed LNI 192.168.203.106@tcp134 [13262.659159] LNet: Removed LNI 192.168.203.106@tcp135 [13263.280751] Key type .llcrypt unregistered [13263.283276] Key type ._llcrypt unregistered [13284.649560] Key type ._llcrypt registered [13284.655607] Key type .llcrypt registered [13284.790474] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [13298.062567] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13298.781478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13298.926152] alg: No test for adler32 (adler32-zlib) [13300.024589] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [13300.293749] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [13301.935176] Key type lgssc registered [13302.891710] Lustre: Echo OBD driver; http://www.lustre.org/ [13336.450784] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:21:15 (1789140075) [13351.696133] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [13353.948173] Key type lgssc unregistered [13354.231519] LNet: 471916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13354.237370] LNetError: 471916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13354.254056] LNet: Removed LNI 192.168.203.106@tcp [13355.051393] Key type .llcrypt unregistered [13355.054757] Key type ._llcrypt unregistered [13369.157397] Key type ._llcrypt registered [13369.160414] Key type .llcrypt registered [13369.279856] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13370.109838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13370.140307] alg: No test for adler32 (adler32-zlib) [13371.256371] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [13371.566476] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [13373.271197] Key type lgssc registered [13374.396934] Lustre: Echo OBD driver; http://www.lustre.org/ [13382.349917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13382.385369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.594160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13383.615699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13383.680981] Lustre: lustre-MDT0000: new disk, initializing [13383.761122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13383.778478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13387.093909] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13396.113537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13396.223413] Lustre: 473107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13396.283147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13396.287234] Lustre: Skipped 1 previous similar message [13396.350270] Lustre: lustre-MDT0001: new disk, initializing [13396.424192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13396.470101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13396.478044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13400.890409] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13409.461083] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13414.359919] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13420.543375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13420.850512] Lustre: lustre-OST0000: new disk, initializing [13420.858747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13420.871260] Lustre: 475077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13420.997657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13426.585917] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13430.859865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13430.872628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13430.995052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13436.192768] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13440.586741] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lnet_if_list [13446.036523] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13446.079083] LNet: Added LNI 192.168.203.106@tcp10 [8/256/0/180] [13452.393674] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13452.421703] LNet: Added LNI 192.168.203.106@tcp11 [8/256/0/180] [13456.711062] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13456.749127] LNet: Added LNI 192.168.203.106@tcp12 [8/256/0/180] [13461.016883] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13461.041168] LNet: Added LNI 192.168.203.106@tcp13 [8/256/0/180] [13466.110198] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13466.137216] LNet: Added LNI 192.168.203.106@tcp14 [8/256/0/180] [13471.067455] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13471.096385] LNet: Added LNI 192.168.203.106@tcp15 [8/256/0/180] [13475.281706] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13475.306798] LNet: Added LNI 192.168.203.106@tcp16 [8/256/0/180] [13479.617363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13479.637810] LNet: Added LNI 192.168.203.106@tcp17 [8/256/0/180] [13484.429637] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13484.454591] LNet: Added LNI 192.168.203.106@tcp18 [8/256/0/180] [13490.969744] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13491.031268] LNet: Added LNI 192.168.203.106@tcp19 [8/256/0/180] [13497.156677] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13497.205681] LNet: Added LNI 192.168.203.106@tcp20 [8/256/0/180] [13504.132712] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13504.186850] LNet: Added LNI 192.168.203.106@tcp21 [8/256/0/180] [13509.070991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13509.099118] LNet: Added LNI 192.168.203.106@tcp22 [8/256/0/180] [13514.406786] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13514.432353] LNet: Added LNI 192.168.203.106@tcp23 [8/256/0/180] [13519.815416] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13519.867844] LNet: Added LNI 192.168.203.106@tcp24 [8/256/0/180] [13525.452455] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13525.492461] LNet: Added LNI 192.168.203.106@tcp25 [8/256/0/180] [13531.425955] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13531.461041] LNet: Added LNI 192.168.203.106@tcp26 [8/256/0/180] [13537.362441] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13537.405664] LNet: Added LNI 192.168.203.106@tcp27 [8/256/0/180] [13543.476381] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13543.501366] LNet: Added LNI 192.168.203.106@tcp28 [8/256/0/180] [13549.483778] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13549.546307] LNet: Added LNI 192.168.203.106@tcp29 [8/256/0/180] [13556.184702] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13556.219872] LNet: Added LNI 192.168.203.106@tcp30 [8/256/0/180] [13562.384302] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13562.449957] LNet: Added LNI 192.168.203.106@tcp31 [8/256/0/180] [13568.999567] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13569.030590] LNet: Added LNI 192.168.203.106@tcp32 [8/256/0/180] [13576.338992] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13576.368591] LNet: Added LNI 192.168.203.106@tcp33 [8/256/0/180] [13583.564164] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13583.609383] LNet: Added LNI 192.168.203.106@tcp34 [8/256/0/180] [13591.107920] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13591.139712] LNet: Added LNI 192.168.203.106@tcp35 [8/256/0/180] [13598.073041] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13598.110973] LNet: Added LNI 192.168.203.106@tcp36 [8/256/0/180] [13605.906566] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13605.942479] LNet: Added LNI 192.168.203.106@tcp37 [8/256/0/180] [13613.531773] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13613.585690] LNet: Added LNI 192.168.203.106@tcp38 [8/256/0/180] [13620.609741] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13620.654026] LNet: Added LNI 192.168.203.106@tcp39 [8/256/0/180] [13628.317700] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13628.358703] LNet: Added LNI 192.168.203.106@tcp40 [8/256/0/180] [13636.437898] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13636.475790] LNet: Added LNI 192.168.203.106@tcp41 [8/256/0/180] [13643.840326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13643.871452] LNet: Added LNI 192.168.203.106@tcp42 [8/256/0/180] [13651.367561] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13651.424166] LNet: Added LNI 192.168.203.106@tcp43 [8/256/0/180] [13658.717904] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13658.758838] LNet: Added LNI 192.168.203.106@tcp44 [8/256/0/180] [13667.416763] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13667.452953] LNet: Added LNI 192.168.203.106@tcp45 [8/256/0/180] [13677.198830] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13677.222225] LNet: Added LNI 192.168.203.106@tcp46 [8/256/0/180] [13686.558725] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13686.623843] LNet: Added LNI 192.168.203.106@tcp47 [8/256/0/180] [13695.741794] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13695.789468] LNet: Added LNI 192.168.203.106@tcp48 [8/256/0/180] [13704.345715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13704.399057] LNet: Added LNI 192.168.203.106@tcp49 [8/256/0/180] [13712.073159] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13712.110160] LNet: Added LNI 192.168.203.106@tcp50 [8/256/0/180] [13731.300775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13731.317910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13731.337298] Lustre: Skipped 1 previous similar message [13731.359856] Lustre: Skipped 2 previous similar messages [13736.418128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13736.441557] Lustre: Skipped 2 previous similar messages [13736.922939] Lustre: server umount lustre-MDT0000 complete [13740.587805] LustreError: 473099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140481 with bad export cookie 8319931379804533168 [13740.594649] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13740.604465] LustreError: 473099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13740.922842] Lustre: server umount lustre-MDT0001 complete [13757.933732] Lustre: 472379:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140482/real 1789140482] req@0000000084fb858a x1876049382248064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13757.971506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13757.988757] Lustre: Skipped 1 previous similar message [13759.187610] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13762.015281] Lustre: 472381:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140486/real 1789140486] req@0000000093daea03 x1876049382248192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13763.104223] Lustre: 472381:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140487/real 1789140487] req@00000000cbe2de9b x1876049382248320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13766.111546] Lustre: 472379:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140491/real 1789140491] req@00000000768b54d2 x1876049382248576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13768.594572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13771.295873] Lustre: 472379:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140496/real 1789140496] req@00000000f443c788 x1876049382248704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789140512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13785.631532] LustreError: 472378:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6ba22e2 x1876049382249472/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13786.212057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13787.233943] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13790.305977] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13792.744880] LustreError: 475071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.775317] LustreError: 475071:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13797.856891] LustreError: 475072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13798.769570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13799.174834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13802.281186] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13804.419264] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13812.594041] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13819.072708] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13838.305767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13838.327341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13838.339688] Lustre: Skipped 1 previous similar message [13843.430738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13844.246934] Lustre: server umount lustre-OST0000 complete [13848.546299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13848.555670] Lustre: Skipped 1 previous similar message [13848.561989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13848.565918] Lustre: Skipped 2 previous similar messages [13858.784241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13858.795579] Lustre: Skipped 3 previous similar messages [13862.367136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13862.580656] Lustre: server umount lustre-MDT0000 complete [13863.906577] LustreError: 499318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13863.928951] LustreError: 499318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13865.992735] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140606 with bad export cookie 8319931379804535814 [13865.999105] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13866.006299] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13866.314918] Lustre: server umount lustre-MDT0001 complete [13875.746613] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [13878.506887] Key type lgssc unregistered [13878.904544] LNet: 502121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13878.924673] LNetError: 502121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13878.947384] LNet: Removed LNI 192.168.203.106@tcp [13878.949929] LNet: Removed LNI 192.168.203.106@tcp10 [13878.961882] LNet: Removed LNI 192.168.203.106@tcp11 [13878.967703] LNet: Removed LNI 192.168.203.106@tcp12 [13878.970762] LNet: Removed LNI 192.168.203.106@tcp13 [13878.984529] LNet: Removed LNI 192.168.203.106@tcp14 [13878.989146] LNet: Removed LNI 192.168.203.106@tcp15 [13878.993880] LNet: Removed LNI 192.168.203.106@tcp16 [13879.004357] LNet: Removed LNI 192.168.203.106@tcp17 [13879.009138] LNet: Removed LNI 192.168.203.106@tcp18 [13879.017145] LNet: Removed LNI 192.168.203.106@tcp19 [13879.025030] LNet: Removed LNI 192.168.203.106@tcp20 [13879.030836] LNet: Removed LNI 192.168.203.106@tcp21 [13879.033421] LNet: Removed LNI 192.168.203.106@tcp22 [13879.041568] LNet: Removed LNI 192.168.203.106@tcp23 [13879.050345] LNet: Removed LNI 192.168.203.106@tcp24 [13879.053949] LNet: Removed LNI 192.168.203.106@tcp25 [13879.062487] LNet: Removed LNI 192.168.203.106@tcp26 [13879.081646] LNet: Removed LNI 192.168.203.106@tcp27 [13879.083826] LNet: Removed LNI 192.168.203.106@tcp28 [13879.092832] LNet: Removed LNI 192.168.203.106@tcp29 [13879.098978] LNet: Removed LNI 192.168.203.106@tcp30 [13879.107081] LNet: Removed LNI 192.168.203.106@tcp31 [13879.112889] LNet: Removed LNI 192.168.203.106@tcp32 [13879.119505] LNet: Removed LNI 192.168.203.106@tcp33 [13879.124444] LNet: Removed LNI 192.168.203.106@tcp34 [13879.131076] LNet: Removed LNI 192.168.203.106@tcp35 [13879.137117] LNet: Removed LNI 192.168.203.106@tcp36 [13879.141348] LNet: Removed LNI 192.168.203.106@tcp37 [13879.143470] LNet: Removed LNI 192.168.203.106@tcp38 [13879.153995] LNet: Removed LNI 192.168.203.106@tcp39 [13879.158379] LNet: Removed LNI 192.168.203.106@tcp40 [13879.162080] LNet: Removed LNI 192.168.203.106@tcp41 [13879.166811] LNet: Removed LNI 192.168.203.106@tcp42 [13879.169659] LNet: Removed LNI 192.168.203.106@tcp43 [13879.173422] LNet: Removed LNI 192.168.203.106@tcp44 [13879.176707] LNet: Removed LNI 192.168.203.106@tcp45 [13879.179195] LNet: Removed LNI 192.168.203.106@tcp46 [13879.182758] LNet: Removed LNI 192.168.203.106@tcp47 [13879.185189] LNet: Removed LNI 192.168.203.106@tcp48 [13879.188416] LNet: Removed LNI 192.168.203.106@tcp49 [13879.197879] LNet: Removed LNI 192.168.203.106@tcp50 [13879.962687] Key type .llcrypt unregistered [13879.966098] Key type ._llcrypt unregistered [13899.242746] Key type ._llcrypt registered [13899.245967] Key type .llcrypt registered [13899.358722] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [13914.205808] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [13915.060075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13915.112297] alg: No test for adler32 (adler32-zlib) [13916.193781] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [13916.411569] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [13918.071186] Key type lgssc registered [13919.213993] Lustre: Echo OBD driver; http://www.lustre.org/ [13946.666837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13946.695933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13947.873967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13947.901297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13947.996668] Lustre: lustre-MDT0000: new disk, initializing [13948.096489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13948.122184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13951.478993] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13959.944300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13960.089381] Lustre: 505320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13960.144102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13960.151084] Lustre: Skipped 1 previous similar message [13960.255237] Lustre: lustre-MDT0001: new disk, initializing [13960.342557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13960.382914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13960.395957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13963.880802] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13971.836828] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13975.864267] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13981.612504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13981.734394] Lustre: *** cfs_fail_loc=90f, val=0*** [13981.778333] Lustre: lustre-OST0000: new disk, initializing [13981.780708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13981.785041] Lustre: 507335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13981.856263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13987.394796] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [13987.898463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13987.912309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13988.086750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13996.321455] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14003.680558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14003.688674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14003.703091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14003.703198] Lustre: Skipped 1 previous similar message [14003.720655] Lustre: Skipped 1 previous similar message [14008.641918] Lustre: server umount lustre-OST0000 complete [14013.935375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14013.938284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14013.949723] Lustre: Skipped 1 previous similar message [14013.964960] Lustre: Skipped 1 previous similar message [14018.482974] Lustre: server umount lustre-MDT0000 complete [14021.694695] LustreError: 505311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140762 with bad export cookie 1737294434368174994 [14021.701697] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14021.708904] LustreError: 505311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14021.977172] Lustre: server umount lustre-MDT0001 complete [14030.713701] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14032.865058] Key type lgssc unregistered [14033.157466] LNet: 509244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14033.165375] LNetError: 509244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14033.179455] LNet: Removed LNI 192.168.203.106@tcp [14033.998148] Key type .llcrypt unregistered [14034.000888] Key type ._llcrypt unregistered [14053.775706] Key type ._llcrypt registered [14053.778201] Key type .llcrypt registered [14053.872493] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14054.915443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14054.963316] alg: No test for adler32 (adler32-zlib) [14056.125678] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [14056.390294] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14058.047965] Key type lgssc registered [14059.110490] Lustre: Echo OBD driver; http://www.lustre.org/ [14065.997790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14066.023045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14067.477750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14071.964646] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14077.924814] LustreError: 510515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14077.941079] LustreError: 510515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14079.293763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14079.639173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14083.463582] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14091.566778] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14097.780384] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14105.669154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14106.231416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14110.968297] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14117.365259] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14131.343426] loop0: detected capacity change from 0 to 204800000 [14131.500588] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14131.513632] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14131.525353] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14136.428867] loop0: detected capacity change from 0 to 204800000 [14136.572140] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14136.586197] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14136.597129] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14140.984837] loop0: detected capacity change from 0 to 204800000 [14147.153843] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14147.348840] Lustre: fs15246-OST0000: new disk, initializing [14147.352133] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14147.358929] Lustre: 514112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14147.480792] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14154.616271] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14163.783469] loop1: detected capacity change from 0 to 204800000 [14168.053166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14168.143452] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14168.191624] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14168.255365] Lustre: fs15246-MDT0000: new disk, initializing [14168.378618] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14168.403400] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14168.480321] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14169.506221] Lustre: 515233:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cfb86d8d x1876050100402816/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789140926 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14171.500142] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14178.599419] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14178.606631] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14178.631797] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14190.567969] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14190.583586] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14193.149905] Lustre: server umount fs15246-MDT0000 complete [14195.687523] LustreError: 510511:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14200.619936] Lustre: server umount fs15246-OST0000 complete [14211.045050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14211.053472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14211.057461] Lustre: Skipped 1 previous similar message [14216.163723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14216.174986] Lustre: Skipped 1 previous similar message [14221.285285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14224.351135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14224.520242] Lustre: server umount lustre-OST0000 complete [14231.523267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14231.532279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14231.545272] Lustre: Skipped 2 previous similar messages [14231.550162] Lustre: Skipped 1 previous similar message [14241.762911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14241.766836] Lustre: Skipped 4 previous similar messages [14243.295125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14243.488608] Lustre: server umount lustre-MDT0000 complete [14246.881642] LustreError: 515047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14246.911171] LustreError: 515047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14247.025168] LustreError: 510496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140987 with bad export cookie 12076686345981834947 [14247.035138] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14247.044888] LustreError: 510496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14247.523862] Lustre: server umount lustre-MDT0001 complete [14256.769197] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14260.010320] Key type lgssc unregistered [14260.330081] LNet: 518336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14260.343084] LNetError: 518336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14260.357054] LNet: Removed LNI 192.168.203.106@tcp [14261.055521] Key type .llcrypt unregistered [14261.058157] Key type ._llcrypt unregistered [14276.232803] Key type ._llcrypt registered [14276.236974] Key type .llcrypt registered [14276.311260] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14277.113119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14277.124292] alg: No test for adler32 (adler32-zlib) [14278.155553] Lustre: Lustre: Build Version: 2.17.57_137_gfce00d1 [14278.281414] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [14279.895160] Key type lgssc registered [14280.986785] Lustre: Echo OBD driver; http://www.lustre.org/ [14314.009688] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [14322.991504] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14355.770817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14355.784641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14357.002699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14357.028723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14357.167706] Lustre: lustre-MDT0000: new disk, initializing [14357.337283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14357.366853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14362.187762] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14371.775480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.921719] Lustre: 522802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14371.975945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14371.981831] Lustre: Skipped 1 previous similar message [14372.051208] Lustre: lustre-MDT0001: new disk, initializing [14372.165214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14372.221783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14372.236101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14376.509705] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14386.141882] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14390.990950] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14396.752298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14396.971734] Lustre: lustre-OST0000: new disk, initializing [14396.974931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14396.980532] Lustre: 524767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14397.053156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14402.299971] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [14403.111451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14403.119173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14403.234236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14410.272377] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14413.857475] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14414.035500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14418.069128] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14418.244907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14423.524551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14423.530107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14423.542963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14424.037123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14424.042222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14429.167763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14429.188371] Lustre: Skipped 2 previous similar messages [14434.276216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14434.280371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14434.299066] Lustre: Skipped 1 previous similar message [14434.399844] Lustre: server umount lustre-OST0000 complete [14438.885136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14438.896530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14438.912875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14444.658491] Lustre: server umount lustre-MDT0000 complete [14447.832411] LustreError: 522792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141188 with bad export cookie 14782200289102110754 [14447.854469] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14448.127788] Lustre: server umount lustre-MDT0001 complete [14470.516437] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [14481.271529] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [14507.547370] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14287 sec ========= 11:40:46 (1789141246) [14509.589605] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:40:48 (1789141248) === [14514.058558] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:40:52 (1789141252) === [14556.485304] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [14559.748737] Key type lgssc unregistered [14560.113817] LNet: 531416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14560.128151] LNetError: 531416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14560.158436] LNet: Removed LNI 192.168.203.106@tcp [14561.097269] Key type .llcrypt unregistered [14561.099598] Key type ._llcrypt unregistered