[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499207151 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003126] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007887] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010127] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.022023] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027238] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028199] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031097] *** VALIDATE tmpfs *** [ 0.033063] *** VALIDATE proc *** [ 0.034143] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043101] debug: unmapping init [mem 0xffffffff8a259000-0xffffffff8a260fff] [ 0.045196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052011] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054275] rcu: Hierarchical SRCU implementation. [ 0.056418] smp: Bringing up secondary CPUs ... [ 0.057538] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.065125] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068019] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.243879] node 0 deferred pages initialised in 173ms [ 0.249262] devtmpfs: initialized [ 0.250412] x86/mm: Memory block size: 128MB [ 0.253000] gcov: version magic: 0x41383552 [ 0.256439] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.257110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258393] pinctrl core: initialized pinctrl subsystem [ 0.259276] [ 0.260013] ************************************************************* [ 0.261020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262253] ** ** [ 0.263019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264021] ** ** [ 0.265017] ** This means that this kernel is built to expose internal ** [ 0.266020] ** IOMMU data structures, which may compromise security on ** [ 0.267016] ** your system. ** [ 0.268021] ** ** [ 0.269017] ** If you see this message and you are not debugging the ** [ 0.270020] ** kernel, report this immediately to your vendor! ** [ 0.271019] ** ** [ 0.272020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273018] ************************************************************* [ 0.275000] NET: Registered protocol family 16 [ 0.277601] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284155] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288097] cpuidle: using governor menu [ 0.291020] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.293671] PCI: Using configuration type 1 for base access [ 0.296159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.309036] cryptd: max_cpu_qlen set to 1000 [ 0.311573] ACPI: Added _OSI(Module Device) [ 0.313019] ACPI: Added _OSI(Processor Device) [ 0.314012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329010] ACPI: Interpreter enabled [ 0.330128] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332016] ACPI: Using IOAPIC for interrupt routing [ 0.334143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337522] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351155] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366706] acpiphp: Slot [2] registered [ 0.368170] acpiphp: Slot [5] registered [ 0.370169] acpiphp: Slot [6] registered [ 0.371132] acpiphp: Slot [7] registered [ 0.373154] acpiphp: Slot [8] registered [ 0.375145] acpiphp: Slot [9] registered [ 0.376267] acpiphp: Slot [10] registered [ 0.378150] acpiphp: Slot [3] registered [ 0.380190] acpiphp: Slot [4] registered [ 0.382122] acpiphp: Slot [11] registered [ 0.383147] acpiphp: Slot [12] registered [ 0.384139] acpiphp: Slot [13] registered [ 0.385281] acpiphp: Slot [14] registered [ 0.387183] acpiphp: Slot [15] registered [ 0.389136] acpiphp: Slot [16] registered [ 0.390114] acpiphp: Slot [17] registered [ 0.392181] acpiphp: Slot [18] registered [ 0.394242] acpiphp: Slot [19] registered [ 0.396296] acpiphp: Slot [20] registered [ 0.397165] acpiphp: Slot [21] registered [ 0.399193] acpiphp: Slot [22] registered [ 0.401096] acpiphp: Slot [23] registered [ 0.403176] acpiphp: Slot [24] registered [ 0.405260] acpiphp: Slot [25] registered [ 0.407181] acpiphp: Slot [26] registered [ 0.409122] acpiphp: Slot [27] registered [ 0.411141] acpiphp: Slot [28] registered [ 0.412250] acpiphp: Slot [29] registered [ 0.414469] acpiphp: Slot [30] registered [ 0.417130] acpiphp: Slot [31] registered [ 0.418061] PCI host bridge to bus 0000:00 [ 0.420046] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424178] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.427022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.430036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.435047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.437254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.441126] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.446497] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.456905] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.461554] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.468028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.472035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.476491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.479963] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.486135] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.490000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.496027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.511022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.519029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.525777] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.539026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.554024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.582078] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.605017] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.615032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.627023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.655018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.667043] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.675018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.687019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.720018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.730886] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.738015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.747015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.766016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.781834] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.801018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.809015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.835017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.867161] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.893021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.901065] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.923023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.938549] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.940382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.943434] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.945357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.947302] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.952209] iommu: Default domain type: Passthrough [ 0.954518] SCSI subsystem initialized [ 0.956195] ACPI: bus type USB registered [ 0.957135] usbcore: registered new interface driver usbfs [ 0.959096] usbcore: registered new interface driver hub [ 0.962092] usbcore: registered new device driver usb [ 0.963157] pps_core: LinuxPPS API ver. 1 registered [ 0.966014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.970089] PTP clock support registered [ 0.972400] EDAC MC: Ver: 3.0.0 [ 0.975322] PCI: Using ACPI for IRQ routing [ 0.977864] NetLabel: Initializing [ 0.979013] NetLabel: domain hash size = 128 [ 0.981010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.982090] NetLabel: unlabeled traffic allowed by default [ 0.984303] vgaarb: loaded [ 0.986082] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.987009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.994146] clocksource: Switched to clocksource kvm-clock [ 1.112165] VFS: Disk quotas dquot_6.6.0 [ 1.113731] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.116568] *** VALIDATE ramfs *** [ 1.117903] *** VALIDATE hugetlbfs *** [ 1.119360] pnp: PnP ACPI init [ 1.122933] pnp: PnP ACPI: found 6 devices [ 1.143724] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.146674] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.148267] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.149884] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.151863] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.153683] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.155898] NET: Registered protocol family 2 [ 1.157865] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.161867] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.164679] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.170060] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.173420] TCP: Hash tables configured (established 65536 bind 65536) [ 1.175582] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.177969] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.180304] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.182809] NET: Registered protocol family 1 [ 1.185119] RPC: Registered named UNIX socket transport module. [ 1.186764] RPC: Registered udp transport module. [ 1.190417] RPC: Registered tcp transport module. [ 1.191832] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.194147] NET: Registered protocol family 44 [ 1.196127] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.198860] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.201051] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.203540] PCI: CLS 0 bytes, default 64 [ 1.205674] Unpacking initramfs... [ 2.692838] debug: unmapping init [mem 0xffff8ea1bcc54000-0xffff8ea1bffbffff] [ 2.697654] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.700458] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.703873] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 3.256412] Initialise system trusted keyrings [ 3.258269] Key type blacklist registered [ 3.261581] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.271070] zbud: loaded [ 3.274037] *** VALIDATE nfs *** [ 3.275993] *** VALIDATE nfs4 *** [ 3.277683] pstore: using deflate compression [ 3.281713] Platform Keyring initialized [ 3.391517] NET: Registered protocol family 38 [ 3.393437] Key type asymmetric registered [ 3.395254] Asymmetric key parser 'x509' registered [ 3.397426] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.400586] io scheduler mq-deadline registered [ 3.402328] io scheduler kyber registered [ 3.403936] io scheduler bfq registered [ 3.406082] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.409477] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.412816] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.416045] ACPI: Power Button [PWRF] [ 3.421526] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.428898] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.461590] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.485687] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.528024] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.583456] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.646628] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.664468] Non-volatile memory driver v1.3 [ 3.668730] Linux agpgart interface v0.103 [ 3.770468] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.777218] vda: detected capacity change from 0 to 71438336 [ 3.838559] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.841408] vdb: detected capacity change from 0 to 1073741824 [ 3.900275] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.913179] vdc: detected capacity change from 0 to 2621440000 [ 3.982474] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.998478] vdd: detected capacity change from 0 to 2621440000 [ 4.044593] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.052290] vde: detected capacity change from 0 to 4294967296 [ 4.098168] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.103840] vdf: detected capacity change from 0 to 4294967296 [ 4.130623] libphy: Fixed MDIO Bus: probed [ 4.139215] usbcore: registered new interface driver usbserial_generic [ 4.150296] usbserial: USB Serial support registered for generic [ 4.162136] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.169951] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.178054] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.184794] mousedev: PS/2 mouse device common for all mice [ 4.188265] rtc_cmos 00:05: RTC can wake from S4 [ 4.191730] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.192861] rtc_cmos 00:05: registered as rtc0 [ 4.211221] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.226744] intel_pstate: CPU model not supported [ 4.232035] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.248854] hid: raw HID events driver (C) Jiri Kosina [ 4.250956] usbcore: registered new interface driver usbhid [ 4.261453] usbhid: USB HID core driver [ 4.264028] drop_monitor: Initializing network drop monitor service [ 4.267082] Initializing XFRM netlink socket [ 4.271768] NET: Registered protocol family 10 [ 4.275839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.283641] Segment Routing with IPv6 [ 4.285874] NET: Registered protocol family 17 [ 4.290063] mpls_gso: MPLS GSO support [ 4.310419] RAS: Correctable Errors collector initialized. [ 4.313953] AVX version of gcm_enc/dec engaged. [ 4.317450] AES CTR mode by8 optimization enabled [ 4.475091] sched_clock: Marking stable (4475067610, 0)->(5453001517, -977933907) [ 4.480958] registered taskstats version 1 [ 4.495142] Loading compiled-in X.509 certificates [ 4.497159] zswap: loaded using pool lzo/zbud [ 4.594459] Key type big_key registered [ 4.662948] Key type encrypted registered [ 4.666234] ima: No TPM chip found, activating TPM-bypass! [ 4.680462] ima: Allocated hash algorithm: sha1 [ 4.687981] ima: No architecture policies found [ 4.698891] evm: Initialising EVM extended attributes: [ 4.709124] evm: security.selinux [ 4.710515] evm: security.ima [ 4.711627] evm: security.capability [ 4.722354] evm: HMAC attrs: 0x1 [ 4.729833] rtc_cmos 00:05: setting system clock to 2026-06-09 17:52:45 UTC (1781027565) [ 4.752342] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 4.769147] debug: unmapping init [mem 0xffffffff89f82000-0xffffffff8a258fff] [ 4.781222] Write protecting the kernel read-only data: 28672k [ 4.803491] debug: unmapping init [mem 0xffffffff88603000-0xffffffff887fffff] [ 4.808807] debug: unmapping init [mem 0xffffffff88f14000-0xffffffff88ffffff] [ 4.943567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.976979] systemd[1]: Detected virtualization kvm. [ 4.991091] systemd[1]: Detected architecture x86-64. [ 4.993943] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.069340] systemd[1]: No hostname configured. [ 5.072381] systemd[1]: Set hostname to . [ 5.076779] random: systemd: uninitialized urandom read (16 bytes read) [ 5.079346] systemd[1]: Initializing machine ID from random generator. [ 5.272245] random: ln: uninitialized urandom read (6 bytes read) [ 5.547129] random: systemd: uninitialized urandom read (16 bytes read) [ 5.558967] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.582891] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.602783] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.475215] device-mapper: uevent: version 1.0.3 [ 7.479446] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.386900] virtio_net virtio0 ens2: renamed from eth0 [ 9.495846] random: fast init done [ 9.773352] scsi host0: ata_piix [ 9.857228] scsi host1: ata_piix [ 9.862107] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.866067] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.243500] random: crng init done [ 15.247558] random: 7 urandom warning(s) missed due to ratelimiting [ 18.039917] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.623747] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.938432] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.889299] SELinux: Disabled at runtime. [ 25.049903] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.063240] systemd[1]: Detected virtualization kvm. [ 25.066251] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.112557] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.130419] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.151651] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.164871] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.175218] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.209816] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.260094] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ 27.921967] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.038677] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.216413] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.274395] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.048557] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.224608] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 37.616253] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 38.690188] NFS: Registering the id_resolver key type [ 38.702968] Key type id_resolver registered [ 38.704793] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 106.010551] libcfs: loading out-of-tree module taints kernel. [ 106.064986] Key type ._llcrypt registered [ 106.067959] Key type .llcrypt registered [ 106.231485] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 123.989567] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 125.324951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.337504] alg: No test for adler32 (adler32-zlib) [ 126.666284] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 127.608127] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 129.312205] Key type lgssc registered [ 130.904650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.043741] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.838728] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 195.670910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.718767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.308871] hrtimer: interrupt took 6696516 ns [ 197.046906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.083949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.165365] Lustre: lustre-MDT0000: new disk, initializing [ 197.233747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.248720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.399361] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.306843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.426328] Lustre: 6415:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.470429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.474023] Lustre: Skipped 1 previous similar message [ 216.566293] Lustre: lustre-MDT0001: new disk, initializing [ 216.664912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.717874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.721888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.155468] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.050937] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.185067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.524271] Lustre: lustre-OST0000: new disk, initializing [ 237.534282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.542610] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.602642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.165473] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.359978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.370780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.466976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.121314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.383917] Lustre: lustre-OST0001: new disk, initializing [ 261.390845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.399783] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.521673] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.226164] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.334866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.348374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.433770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.282413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.177431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.627922] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.668168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 303.950516] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 307.079136] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 310.293107] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 312.331869] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 13:57:50 EDT 2026 [ 329.230692] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 330.952247] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.664923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 349.677075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.688881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.179051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.194333] Lustre: Skipped 1 previous similar message [ 350.202539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.207539] Lustre: Skipped 3 previous similar messages [ 353.244684] Lustre: server umount lustre-MDT0000 complete [ 356.755556] LustreError: 10423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781027917 with bad export cookie 142734559360837284 [ 356.760522] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.772947] LustreError: 10423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 360.422349] LustreError: 9865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.424429] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.425077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.436813] LustreError: 9865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 360.458900] Lustre: Skipped 2 previous similar messages [ 361.441829] LustreError: 6422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.462990] LustreError: 6422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.560587] LustreError: 6423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.560953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.582149] LustreError: 6423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.605465] Lustre: Skipped 4 previous similar messages [ 371.170513] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 371.648024] Lustre: server umount lustre-MDT0001 complete [ 375.942504] Lustre: server umount lustre-OST0000 complete [ 380.152461] Lustre: server umount lustre-OST0001 complete [ 387.631977] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 398.766474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 446.021234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.481369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 446.513033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 446.619655] Lustre: lustre-MDT0000: new disk, initializing [ 446.730413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.754439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 452.031222] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 465.221812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.419070] Lustre: 14941:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.541397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.549869] Lustre: Skipped 1 previous similar message [ 465.617710] Lustre: lustre-MDT0001: new disk, initializing [ 465.750294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 465.780941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 465.787580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.449545] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 480.676192] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 485.565593] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 494.113344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.447261] Lustre: lustre-OST0000: new disk, initializing [ 494.450279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.458468] Lustre: 16781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 494.562927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.222295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.244653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.498381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 501.923893] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 511.915449] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 516.965069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.186904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 521.882517] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.157511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.329655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.345825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.351707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.362803] Lustre: Skipped 1 previous similar message [ 530.302604] Lustre: server umount lustre-OST0000 complete [ 537.572089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.600565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.603463] Lustre: Skipped 1 previous similar message [ 540.372925] Lustre: server umount lustre-MDT0000 complete [ 542.689296] LustreError: 14953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.712234] LustreError: 14953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 544.605136] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028105 with bad export cookie 142734559360838817 [ 544.606969] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.621693] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.053867] Lustre: server umount lustre-MDT0001 complete [ 550.565183] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:01:48 (1781028108) [ 553.475083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 555.675491] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:01:54 (1781028114) [ 562.495515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.055979] LustreError: 19104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.160609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.439949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 572.385536] LustreError: 19105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.400324] LustreError: 19105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 578.233259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.266109] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 590.742432] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 595.831513] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 603.423205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 603.571049] LustreError: 20794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.585299] LustreError: 20794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 610.499492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 618.408616] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 628.706155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.713738] LustreError: Skipped 1 previous similar message [ 628.716954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.722983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 628.726860] Lustre: Skipped 2 previous similar messages [ 628.762503] Lustre: Skipped 1 previous similar message [ 633.238311] Lustre: server umount lustre-OST0000 complete [ 643.611428] Lustre: server umount lustre-MDT0000 complete [ 644.070777] LustreError: 19099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.095447] LustreError: 19099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 647.795959] LustreError: 19085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028208 with bad export cookie 142734559360839699 [ 647.808454] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.311506] Lustre: server umount lustre-MDT0001 complete [ 660.412130] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 663.812190] Key type lgssc unregistered [ 664.119793] LNet: 22427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.134237] LNetError: 22427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.169170] LNet: Removed LNI 192.168.204.115@tcp [ 665.082314] Key type .llcrypt unregistered [ 665.084781] Key type ._llcrypt unregistered [ 693.078334] Key type ._llcrypt registered [ 693.084658] Key type .llcrypt registered [ 693.233736] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 694.401304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.456312] alg: No test for adler32 (adler32-zlib) [ 695.823328] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 696.127729] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 697.834692] Key type lgssc registered [ 699.180804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 708.664141] loop: module loaded [ 708.807666] loop0: detected capacity change from 0 to 204800000 [ 709.116701] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.127797] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.611608] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.302428] loop0: detected capacity change from 0 to 204800000 [ 716.459874] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.486759] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.511352] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.105236] loop0: detected capacity change from 0 to 204800000 [ 727.960446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 727.980821] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.250861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 729.295035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 729.419362] Lustre: lustre-MDT0000: new disk, initializing [ 729.501954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.540790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 733.870335] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 744.061978] loop1: detected capacity change from 0 to 204800000 [ 749.656377] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.069365] Lustre: lustre-OST07c6: new disk, initializing [ 750.075616] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 750.079588] Lustre: Skipped 1 previous similar message [ 750.083212] Lustre: 25633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 750.153450] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 755.325331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 755.342911] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 755.380134] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 756.827669] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 762.560292] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.569908] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.575324] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.581138] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.586291] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 762.590609] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 777.735759] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 777.747479] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.756587] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.766674] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 777.777065] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 777.791273] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.725658] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 782.737031] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 782.752604] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 782.766892] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.778142] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.785304] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.790514] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 782.795668] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.801606] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 782.806668] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.813101] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.820330] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.061852] Lustre: 24589:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 786.076934] Lustre: 24589:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 786.084471] Lustre: 24589:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 786.094836] Lustre: 24589:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.109596] Lustre: 24589:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.126598] Lustre: 24589:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.134331] Lustre: 24589:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 786.141358] Lustre: 24589:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.147536] Lustre: 24589:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.153067] Lustre: 24589:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.162602] Lustre: 24589:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.171265] Lustre: 24589:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.038116] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 791.059260] Lustre: 24588:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 791.071214] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.079180] Lustre: 24588:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.089549] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.096058] Lustre: 24588:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.104814] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 791.112222] Lustre: 24588:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.118634] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 791.124800] Lustre: 24588:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.130300] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.135588] Lustre: 24588:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.731471] Lustre: 24590:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 802.745051] Lustre: 24590:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 802.755811] Lustre: 24590:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 802.765243] Lustre: 24590:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 802.777801] Lustre: 24590:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 802.790938] Lustre: 24590:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 802.796802] Lustre: 24590:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 802.802945] Lustre: 24590:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 802.808325] Lustre: 24590:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 802.814354] Lustre: 24590:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 802.821183] Lustre: 24590:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 802.834830] Lustre: 24590:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 852.454686] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.461132] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.470096] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 856.100516] Lustre: server umount lustre-OST07c6 complete [ 865.989582] Lustre: server umount lustre-MDT0000 complete [ 881.605433] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 884.416747] Key type lgssc unregistered [ 884.700958] LNet: 28201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.708494] LNetError: 28201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.732764] LNet: Removed LNI 192.168.204.115@tcp [ 885.488810] Key type .llcrypt unregistered [ 885.490612] Key type ._llcrypt unregistered [ 903.736179] Key type ._llcrypt registered [ 903.738273] Key type .llcrypt registered [ 903.842539] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 904.963226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.000183] alg: No test for adler32 (adler32-zlib) [ 906.096652] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 906.340730] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 908.048247] Key type lgssc registered [ 909.196966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.262967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.300263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.821091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.521619] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 928.232417] LustreError: 29457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.253225] LustreError: 29457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 931.590516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.888374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.909174] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 941.853956] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.370804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 950.874577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.152720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.564430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 962.007363] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 963.442515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 976.865624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.870480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.880448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.922608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.925206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.932287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.936792] LustreError: Skipped 1 previous similar message [ 978.963409] Lustre: Skipped 1 previous similar message [ 982.962720] Lustre: server umount lustre-OST0000 complete [ 988.130887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.141372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.149140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.097076] Lustre: server umount lustre-MDT0000 complete [ 994.275198] LustreError: 29458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.290098] LustreError: 29458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 996.862854] LustreError: 29436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028557 with bad export cookie 4196375380575215952 [ 996.866297] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.881601] LustreError: 29436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 997.165540] Lustre: server umount lustre-MDT0001 complete [ 1009.178326] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1011.746800] Key type lgssc unregistered [ 1012.053485] LNet: 32979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.065211] LNetError: 32979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.083930] LNet: Removed LNI 192.168.204.115@tcp [ 1012.772934] Key type .llcrypt unregistered [ 1012.777975] Key type ._llcrypt unregistered [ 1031.955990] Key type ._llcrypt registered [ 1031.958232] Key type .llcrypt registered [ 1032.068528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1032.798687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.836405] alg: No test for adler32 (adler32-zlib) [ 1033.885611] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1034.082314] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1035.744894] Key type lgssc registered [ 1036.590753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.939596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.954784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.299247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.174427] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1054.692943] LustreError: 34233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.716041] LustreError: 34233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1057.358435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.621042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.501881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1067.314972] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.693890] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.179475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.547128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.630144] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1088.284483] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1089.860929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1105.378861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.381623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.402126] Lustre: Skipped 1 previous similar message [ 1109.745826] Lustre: server umount lustre-OST0000 complete [ 1114.083366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1114.090620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.114693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.122934] Lustre: Skipped 1 previous similar message [ 1115.617815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.622991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.784477] Lustre: server umount lustre-MDT0000 complete [ 1120.739118] LustreError: 34234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.748775] LustreError: 34234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1123.247771] LustreError: 34215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028684 with bad export cookie 3481277250705603624 [ 1123.264703] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.579634] Lustre: server umount lustre-MDT0001 complete [ 1133.841563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1136.320514] Key type lgssc unregistered [ 1136.583589] LNet: 37576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.599928] LNetError: 37576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.619895] LNet: Removed LNI 192.168.204.115@tcp [ 1137.493622] Key type .llcrypt unregistered [ 1137.495725] Key type ._llcrypt unregistered [ 1159.105372] Key type ._llcrypt registered [ 1159.109805] Key type .llcrypt registered [ 1159.241965] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1160.229752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1160.337260] alg: No test for adler32 (adler32-zlib) [ 1161.352303] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1161.568673] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1163.209634] Key type lgssc registered [ 1164.049179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1170.530509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1170.543916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.880896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.469181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1182.181540] LustreError: 38976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.197617] LustreError: 38976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1184.309537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.670260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.285809] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1193.991968] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1197.783783] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.183032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.549975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.117994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1216.967858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1217.339249] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1224.162966] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1225.701431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.713834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.735149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.245064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.253984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.264415] Lustre: Skipped 1 previous similar message [ 1229.248888] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1229.260390] Lustre: Skipped 2 previous similar messages [ 1229.552685] Lustre: server umount lustre-MDT0000 complete [ 1232.353872] LustreError: 38977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.368783] LustreError: 38977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1233.441064] LustreError: 38958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028794 with bad export cookie 14401916868281151433 [ 1233.458017] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.463430] LustreError: 38958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1234.363684] LustreError: 40684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.414929] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 1237.473490] LustreError: 38973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.475550] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.856797] Lustre: server umount lustre-MDT0001 complete [ 1269.221058] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1269.572487] Lustre: server umount lustre-OST0000 complete [ 1284.524438] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1286.649707] Key type lgssc unregistered [ 1286.932494] LNet: 42624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.940548] LNetError: 42624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.958398] LNet: Removed LNI 192.168.204.115@tcp [ 1287.790844] Key type .llcrypt unregistered [ 1287.792234] Key type ._llcrypt unregistered [ 1308.750126] Key type ._llcrypt registered [ 1308.752809] Key type .llcrypt registered [ 1308.867550] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1309.784427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.872919] alg: No test for adler32 (adler32-zlib) [ 1311.050091] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1311.278448] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1312.960694] Key type lgssc registered [ 1313.971381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1321.139995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1321.155133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.610422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.818285] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1333.226802] LustreError: 44003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.240069] LustreError: 44003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1334.144745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.455643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1337.845755] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1343.451641] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1346.579286] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1352.192778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.423299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.952941] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1362.200710] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1367.792883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1373.157866] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1373.159988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1373.176378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.559055] Lustre: server umount lustre-OST0000 complete [ 1393.635691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.643136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.646088] Lustre: Skipped 2 previous similar messages [ 1393.660815] Lustre: Skipped 3 previous similar messages [ 1394.658272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.666507] Lustre: Skipped 1 previous similar message [ 1399.664445] Lustre: server umount lustre-MDT0000 complete [ 1399.779285] LustreError: 44004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.801433] LustreError: 44004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1403.119797] LustreError: 43985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781028963 with bad export cookie 2635522141046391449 [ 1403.125081] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1403.134960] LustreError: 43985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1409.499256] Lustre: server umount lustre-MDT0001 complete [ 1418.352362] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1420.432297] Key type lgssc unregistered [ 1420.657629] LNet: 47500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.661477] LNetError: 47500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.671255] LNet: Removed LNI 192.168.204.115@tcp [ 1421.329666] Key type .llcrypt unregistered [ 1421.331910] Key type ._llcrypt unregistered [ 1438.741705] Key type ._llcrypt registered [ 1438.744327] Key type .llcrypt registered [ 1438.872039] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1439.789846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.813581] alg: No test for adler32 (adler32-zlib) [ 1440.884697] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1441.108766] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1442.776488] Key type lgssc registered [ 1443.726076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1449.817775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1449.835140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.199229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.312836] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1460.709226] LustreError: 48880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.728235] LustreError: 48880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1462.421799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.730166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.381413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1471.817950] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1475.309128] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1481.061652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.312292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.525679] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1492.169837] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.667474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1502.164786] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1505.011882] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1516.583486] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0914h17m55s [ 1517.743726] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1519.089971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1519.099135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.115286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.167333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.175596] Lustre: Skipped 5 previous similar messages [ 1525.389169] Lustre: server umount lustre-MDT0000 complete [ 1526.249333] LustreError: 48875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.269363] LustreError: 48875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1528.076904] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1531.775897] LustreError: 48875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.790557] LustreError: 48875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1532.947955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.084453] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1533.265211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.306703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1534.334369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1536.209843] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1539.726487] Lustre: DEBUG MARKER: Wait for df (37089) ... [ 1540.874921] Lustre: DEBUG MARKER: done [ 1544.163156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.174079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.187117] Lustre: Skipped 2 previous similar messages [ 1544.192438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.197416] Lustre: Skipped 1 previous similar message [ 1549.805167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1549.810350] Lustre: Skipped 2 previous similar messages [ 1549.951251] Lustre: server umount lustre-OST0000 complete [ 1554.918453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.920919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1554.922821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.939446] Lustre: Skipped 2 previous similar messages [ 1558.897767] Lustre: server umount lustre-MDT0000 complete [ 1560.035509] LustreError: 48880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.062699] LustreError: 48880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1561.605077] LustreError: 51311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029122 with bad export cookie 10781221490128896737 [ 1561.607805] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.628187] LustreError: 51311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1561.823844] Lustre: server umount lustre-MDT0001 complete [ 1569.455981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1571.579246] Key type lgssc unregistered [ 1571.829282] LNet: 53621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.837752] LNetError: 53621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.861311] LNet: Removed LNI 192.168.204.115@tcp [ 1572.451589] Key type .llcrypt unregistered [ 1572.458738] Key type ._llcrypt unregistered [ 1603.195503] Key type ._llcrypt registered [ 1603.198285] Key type .llcrypt registered [ 1603.280762] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1603.919251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.009730] alg: No test for adler32 (adler32-zlib) [ 1605.036562] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1605.210500] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1606.873791] Key type lgssc registered [ 1607.736265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.954757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1613.967170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.198183] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1615.226780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1615.361978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.581491] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1626.395432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.489443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1626.518376] Lustre: 55791:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1626.739273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.708967] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1634.825636] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1637.708463] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1643.382925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.508103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1643.671086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.435153] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1652.835540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1653.730717] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1659.619306] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0914h20m18s [ 1660.753939] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1666.143749] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1667.929527] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1667.972329] LustreError: 57327:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1688.041193] Lustre: lustre-MDT0000: Client a1a34e67-ac3e-468d-9738-3a12edc7bfa9 (at 192.168.204.15@tcp) reconnecting [ 1708.024162] LustreError: 57327:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1709.822639] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1711.991965] Lustre: DEBUG MARKER: done [ 1719.267223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.295426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.303248] Lustre: Skipped 1 previous similar message [ 1723.244089] Lustre: server umount lustre-OST0000 complete [ 1728.998900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1729.005603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.014430] Lustre: Skipped 1 previous similar message [ 1729.026578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.081965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.088108] Lustre: Skipped 2 previous similar messages [ 1732.408616] Lustre: server umount lustre-MDT0000 complete [ 1735.715052] LustreError: 55782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029296 with bad export cookie 14420990512369913568 [ 1735.733181] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1735.959219] Lustre: server umount lustre-MDT0001 complete [ 1743.930535] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1745.855321] Key type lgssc unregistered [ 1746.069622] LNet: 59906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.073687] LNetError: 59906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.089360] LNet: Removed LNI 192.168.204.115@tcp [ 1746.713197] Key type .llcrypt unregistered [ 1746.715875] Key type ._llcrypt unregistered [ 1778.305881] Key type ._llcrypt registered [ 1778.315977] Key type .llcrypt registered [ 1778.394290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1779.033535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.064236] alg: No test for adler32 (adler32-zlib) [ 1780.052520] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 1780.213841] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1781.872355] Key type lgssc registered [ 1782.601874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1784.809535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.830444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.012865] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1786.028461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1786.124285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.827842] Lustre: Failing over lustre-MDT0000 [ 1787.006625] Lustre: server umount lustre-MDT0000 complete [ 1791.516892] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:22:30 (1781029350) [ 1796.498966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.954283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.320327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1807.665144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.735678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1807.791018] Lustre: 62664:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1808.064293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1811.194154] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1816.380845] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1819.448507] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1824.579514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.730863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1824.930899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.066635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1832.518941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1833.882532] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1838.098390] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1842.660646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.663750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.672883] Lustre: Skipped 1 previous similar message [ 1842.679705] Lustre: Skipped 2 previous similar messages [ 1844.395811] Lustre: server umount lustre-MDT0000 complete [ 1846.792814] LustreError: 62655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029407 with bad export cookie 5908416762549774993 [ 1846.795683] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.800559] LustreError: 62655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1846.985449] Lustre: server umount lustre-MDT0001 complete [ 1849.020874] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1852.277848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1857.705801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.096171] Lustre: 61590:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029408/real 1781029408] req@0000000074649beb x1867544621253888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1864.112444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.120076] Lustre: Skipped 1 previous similar message [ 1867.936163] Lustre: 61591:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029412/real 1781029412] req@000000006883bcb1 x1867544621254016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1871.331442] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x51fee9bdccbe3374 [ 1871.339769] Lustre: MGC192.168.204.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1871.582304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.646302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1872.617119] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029417/real 1781029417] req@0000000070e25f6b x1867544621254400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1872.639886] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1874.679474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1877.729281] Lustre: 61590:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029422/real 1781029422] req@00000000a7a838f8 x1867544621254528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1879.015250] LustreError: 66238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.029407] LustreError: 66238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1881.916125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.257660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.315269] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1885.683758] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1890.573675] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1893.107476] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1898.979969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.981699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.991872] Lustre: Skipped 2 previous similar messages [ 1898.999157] Lustre: Skipped 2 previous similar messages [ 1903.781089] Lustre: server umount lustre-MDT0000 complete [ 1904.099275] LustreError: 66236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.112734] LustreError: 66236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1905.983864] LustreError: 62655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029466 with bad export cookie 5908416762549777268 [ 1905.986886] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.994593] LustreError: 62655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1906.180334] Lustre: server umount lustre-MDT0001 complete [ 1910.571693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1916.664463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.576226] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029469/real 1781029469] req@000000008abed0d7 x1867544621278464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1924.601923] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1924.613150] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.784866] LustreError: 61587:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1b6f260 x1867544621279488/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1931.022591] LustreError: 64382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.077548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.117638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1933.090539] LustreError: 64383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.252357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1937.184115] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781029481/real 1781029481] req@000000009aaa4804 x1867544621279104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781029497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.209789] Lustre: 61588:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1940.450225] LustreError: 68790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.461308] LustreError: 68790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1940.583872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.777399] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1946.099462] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1946.106668] Lustre: Skipped 1 previous similar message [ 1948.416283] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1950.825872] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.257486] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1956.322802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.333422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.341884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.390571] Lustre: server umount lustre-MDT0000 complete [ 1958.369676] LustreError: 64383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.383083] LustreError: 64383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1959.787817] LustreError: 63430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029520 with bad export cookie 5908416762549778647 [ 1959.792813] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.796510] LustreError: 63430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1962.071165] Lustre: server umount lustre-MDT0001 complete [ 1966.289740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1968.434226] Lustre: DEBUG MARKER: files compared the same [ 1979.911448] Lustre: server umount lustre-OST0000 complete [ 1990.165839] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1992.073625] Key type lgssc unregistered [ 1992.294085] LNet: 71923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.303137] LNetError: 71923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.316104] LNet: Removed LNI 192.168.204.115@tcp [ 1992.952153] Key type .llcrypt unregistered [ 1992.955313] Key type ._llcrypt unregistered [ 2008.296621] Key type ._llcrypt registered [ 2008.298888] Key type .llcrypt registered [ 2008.389277] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2008.918296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.945067] alg: No test for adler32 (adler32-zlib) [ 2009.904882] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2010.062290] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2011.672227] Key type lgssc registered [ 2012.407019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.224331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2017.236812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.560676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.034969] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2027.007378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.210989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2029.811159] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2034.324440] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2036.944263] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2041.203709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.411610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.496599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2045.419642] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2049.554421] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2057.188864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2057.190251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.192521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2057.203323] Lustre: Skipped 1 previous similar message [ 2061.879098] Lustre: server umount lustre-OST0000 complete [ 2067.429120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.433127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.438478] Lustre: Skipped 1 previous similar message [ 2067.441387] Lustre: Skipped 1 previous similar message [ 2070.327929] Lustre: server umount lustre-MDT0000 complete [ 2072.547656] LustreError: 73297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.556807] LustreError: 73297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2072.763597] LustreError: 75014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029633 with bad export cookie 16749178221435460260 [ 2072.767702] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.770595] LustreError: 75014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2072.986336] Lustre: server umount lustre-MDT0001 complete [ 2079.355516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2080.901586] Key type lgssc unregistered [ 2081.084680] LNet: 76618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.088913] LNetError: 76618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.099774] LNet: Removed LNI 192.168.204.115@tcp [ 2081.588380] Key type .llcrypt unregistered [ 2081.590707] Key type ._llcrypt unregistered [ 2095.787604] Key type ._llcrypt registered [ 2095.789527] Key type .llcrypt registered [ 2095.857175] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2096.347034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.359798] alg: No test for adler32 (adler32-zlib) [ 2097.317481] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2097.472824] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2099.096575] Key type lgssc registered [ 2099.748043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2104.319567] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2104.335658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.840665] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2124.960310] LustreError: 77971:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 2151.584199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.677061] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2158.301909] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2162.801313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.873304] Lustre: *** cfs_fail_loc=706, val=0*** [ 2163.142278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.193098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2165.497308] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2171.287544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.362581] LustreError: 79304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.380352] LustreError: 79304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2171.531487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.002043] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2178.249872] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.767946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2182.116886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.120286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.125613] Lustre: Skipped 1 previous similar message [ 2187.236412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.241466] Lustre: Skipped 1 previous similar message [ 2188.230791] Lustre: server umount lustre-OST0000 complete [ 2191.842320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.846766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.853558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.857994] Lustre: Skipped 1 previous similar message [ 2196.878517] Lustre: server umount lustre-MDT0000 complete [ 2197.472952] LustreError: 78993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.481760] LustreError: 78993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2199.074613] LustreError: 77979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781029759 with bad export cookie 6005588057869365633 [ 2199.079179] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.085140] LustreError: 77979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2199.316775] Lustre: server umount lustre-MDT0001 complete [ 2205.249430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2206.844418] Key type lgssc unregistered [ 2207.054054] LNet: 81368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.063595] LNetError: 81368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.079074] LNet: Removed LNI 192.168.204.115@tcp [ 2207.534805] Key type .llcrypt unregistered [ 2207.536880] Key type ._llcrypt unregistered [ 2221.420258] Key type ._llcrypt registered [ 2221.422327] Key type .llcrypt registered [ 2221.468759] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2221.912289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.955559] alg: No test for adler32 (adler32-zlib) [ 2222.907967] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2223.045389] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2224.664174] Key type lgssc registered [ 2225.326830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.887626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.899577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.063531] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2238.037414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.357270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.333824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2243.554252] LustreError: 83463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.572807] LustreError: 83463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2246.096746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.279039] LustreError: 83462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.292615] LustreError: 83462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2246.300933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.355881] LustreError: 84239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.372412] LustreError: 84239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2249.306027] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2253.798579] LustreError: 83463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.809390] LustreError: 83463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.705161] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.714048] Lustre: MGS: Client a521e15a-2375-43ce-8264-f82d6acd3ab1 (at 0@lo) reconnecting [ 2254.846332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.887239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2257.062725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2270.177092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2270.182652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.184463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2270.192017] Lustre: Skipped 1 previous similar message [ 2273.905238] Lustre: server umount lustre-OST0000 complete [ 2280.418084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2280.419662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.423537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.437842] Lustre: Skipped 1 previous similar message [ 2280.441421] Lustre: Skipped 1 previous similar message [ 2285.538530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.544849] Lustre: Skipped 1 previous similar message [ 2290.144423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2290.232356] Lustre: server umount lustre-MDT0000 complete [ 2290.662995] LustreError: 83466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.677914] LustreError: 83466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2292.600339] Lustre: server umount lustre-MDT0001 complete [ 2294.806575] Lustre: server umount MGS complete [ 2301.085036] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:31:00 (1781029860) [ 2317.170450] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2322.037327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2341.641073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.002842] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2350.116687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.235764] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2350.267996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2350.321409] Lustre: lustre-MDT0001: new disk, initializing [ 2350.374476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2350.389399] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2350.395324] Lustre: 90714:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2350.401336] LustreError: 90714:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2352.566833] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2359.144697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2359.287884] Lustre: lustre-OST0000: new disk, initializing [ 2359.290886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2359.295548] Lustre: 91512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2359.332944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.302826] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2367.512597] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2369.569354] Lustre: 92222:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000085fba255 x1867545085777152/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781029946 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2370.878903] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2370.889760] Lustre: MGS: Client d7760817-1511-4012-8ba7-a3ae7b5436de (at 0@lo) reconnecting [ 2370.934020] Lustre: 89951:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2370.951156] LustreError: 90700:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2370.957678] LustreError: 90700:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2370.965985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2371.019753] Lustre: lustre-MDT0000: new disk, initializing [ 2371.097747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.116430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2372.193233] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2372.193328] Lustre: 92504:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006059114c x1867545085782784/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781029948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2372.209034] Lustre: 92504:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 2373.049457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2381.289945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2381.296332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2381.338238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2404.322237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.326247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.327890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2404.331988] Lustre: Skipped 1 previous similar message [ 2409.443027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.447362] Lustre: Skipped 2 previous similar messages [ 2409.850600] Lustre: server umount lustre-OST0000 complete [ 2414.561808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.562711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.575820] Lustre: Skipped 1 previous similar message [ 2419.682259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.687328] Lustre: Skipped 3 previous similar messages [ 2426.336147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2426.416078] Lustre: server umount lustre-MDT0000 complete [ 2434.574078] Lustre: server umount lustre-MDT0001 complete [ 2436.461164] Lustre: server umount MGS complete [ 2441.517460] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:33:21 (1781030001) [ 2449.930544] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2451.286169] Key type lgssc unregistered [ 2451.448713] LNet: 95024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.452988] LNetError: 95024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.461321] LNet: Removed LNI 192.168.204.115@tcp [ 2451.830256] Key type .llcrypt unregistered [ 2451.831846] Key type ._llcrypt unregistered [ 2460.377527] Key type ._llcrypt registered [ 2460.379399] Key type .llcrypt registered [ 2460.428957] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2460.961051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.984047] alg: No test for adler32 (adler32-zlib) [ 2461.880437] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2461.997035] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2463.618102] Key type lgssc registered [ 2464.195064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2467.808829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2467.826698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.086800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.049974] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2475.791865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.970512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2477.863405] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2481.095237] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2482.821092] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2486.370117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2486.562968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.192840] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2492.120808] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.736223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2507.239427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.240366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.241901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.242888] LustreError: Skipped 1 previous similar message [ 2507.250207] Lustre: Skipped 1 previous similar message [ 2511.797138] Lustre: server umount lustre-OST0000 complete [ 2516.961750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2516.969901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.978432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.980988] Lustre: Skipped 1 previous similar message [ 2519.891986] Lustre: server umount lustre-MDT0000 complete [ 2521.912132] LustreError: 96186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030082 with bad export cookie 14018906152808391754 [ 2521.914460] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.915711] LustreError: 96963:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2521.915755] LustreError: 96963:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2521.921200] LustreError: 96186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2522.081887] Lustre: server umount lustre-MDT0001 complete [ 2526.824630] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2528.246777] Key type lgssc unregistered [ 2528.421217] LNet: 99545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.429279] LNetError: 99545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.438342] LNet: Removed LNI 192.168.204.115@tcp [ 2528.906595] Key type .llcrypt unregistered [ 2528.908250] Key type ._llcrypt unregistered [ 2536.933992] Key type ._llcrypt registered [ 2536.935635] Key type .llcrypt registered [ 2536.993491] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2537.451727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.461912] alg: No test for adler32 (adler32-zlib) [ 2538.374407] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2538.492786] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2540.096207] Key type lgssc registered [ 2540.680682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2543.528907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.532606] LustreError: 100593:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2543.540190] LustreError: 100600:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2543.543555] LustreError: 100593:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2543.547251] LustreError: 100593:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2543.551349] LustreError: 100593:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2543.554712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.790978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.329650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.460679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2550.229386] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2553.328231] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lsmod [ 2554.521192] LustreError: 101813:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2554.536930] LustreError: 101820:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2554.540745] LustreError: 101813:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 2554.546572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.546859] LustreError: 101813:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2554.555400] LustreError: 101813:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2554.728166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.761510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2558.945381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.953081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.960872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.971265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.974774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.980342] Lustre: Skipped 1 previous similar message [ 2559.992691] Lustre: Skipped 1 previous similar message [ 2562.280434] Lustre: server umount lustre-MDT0000 complete [ 2564.083273] LustreError: 100618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030124 with bad export cookie 4923709303057829650 [ 2564.084744] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.089990] LustreError: 100618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2565.089087] LustreError: 100633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.089882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.097173] LustreError: 100633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2565.105859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2565.111547] Lustre: Skipped 1 previous similar message [ 2566.112391] LustreError: 101069:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b32fb952 x1867545417107328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2566.207450] Lustre: server umount lustre-MDT0001 complete [ 2568.176284] Lustre: server umount lustre-OST0000 complete [ 2571.838788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.037876] LustreError: 102884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.067090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.639085] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2577.377266] LustreError: 102885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.491709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.672434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2579.315926] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2582.132389] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2583.729553] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2586.864445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.033654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.569893] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2592.439273] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2596.264758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2601.445108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.445995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2601.448327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.452151] Lustre: Skipped 1 previous similar message [ 2605.480064] Lustre: server umount lustre-OST0000 complete [ 2608.608817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2608.612122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.618730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.621718] Lustre: Skipped 1 previous similar message [ 2613.365417] Lustre: server umount lustre-MDT0000 complete [ 2614.951416] LustreError: 104595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030175 with bad export cookie 4923709303057830567 [ 2614.956491] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.961939] LustreError: 104595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2615.123290] Lustre: server umount lustre-MDT0001 complete [ 2619.773561] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2621.074194] Key type lgssc unregistered [ 2621.211771] LNet: 106199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.218286] LNetError: 106199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.224463] LNet: Removed LNI 192.168.204.115@tcp [ 2621.626306] Key type .llcrypt unregistered [ 2621.627979] Key type ._llcrypt unregistered [ 2632.629054] Key type ._llcrypt registered [ 2632.631050] Key type .llcrypt registered [ 2632.678597] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2633.063674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.107802] alg: No test for adler32 (adler32-zlib) [ 2633.999120] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2634.096695] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2635.696235] Key type lgssc registered [ 2636.177513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2639.364885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2639.374265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.588994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.101832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2645.936808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.114852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2647.632379] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2650.366396] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2651.900917] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2654.893111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.085673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.375907] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2660.107403] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2666.347704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2672.634970] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2673.612578] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2676.710578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.712633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.713549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.717749] Lustre: Skipped 1 previous similar message [ 2676.726770] LustreError: Skipped 1 previous similar message [ 2681.119658] Lustre: server umount lustre-OST0000 complete [ 2686.945746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2686.945822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.946450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.946454] Lustre: Skipped 1 previous similar message [ 2686.959638] Lustre: Skipped 1 previous similar message [ 2688.730843] Lustre: server umount lustre-MDT0000 complete [ 2690.256927] LustreError: 107556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030251 with bad export cookie 13583803590719183883 [ 2690.262498] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.262817] LustreError: 107556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2690.395721] Lustre: server umount lustre-MDT0001 complete [ 2694.521362] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2695.701757] Key type lgssc unregistered [ 2695.843606] LNet: 111019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.847176] LNetError: 111019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.855603] LNet: Removed LNI 192.168.204.115@tcp [ 2696.158400] Key type .llcrypt unregistered [ 2696.160028] Key type ._llcrypt unregistered [ 2704.103527] Key type ._llcrypt registered [ 2704.106026] Key type .llcrypt registered [ 2704.158854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2704.712956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2704.752301] alg: No test for adler32 (adler32-zlib) [ 2705.655177] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2705.766457] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2707.360079] Key type lgssc registered [ 2707.874505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2711.535699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2711.548910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.796820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.663381] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2719.804048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.031205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.075079] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2725.395439] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2727.124426] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2730.493615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.687898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.343606] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2736.296500] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2737.837272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2753.506071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.510381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.520876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.556216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.557682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.560497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.562544] Lustre: Skipped 1 previous similar message [ 2756.700200] Lustre: server umount lustre-OST0000 complete [ 2760.675721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.678084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.681315] Lustre: Skipped 1 previous similar message [ 2764.452609] Lustre: server umount lustre-MDT0000 complete [ 2765.793701] LustreError: 113301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.801112] LustreError: 113301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2766.045118] LustreError: 114806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030326 with bad export cookie 7668312188084946165 [ 2766.048686] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.051309] LustreError: 114806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2766.190996] Lustre: server umount lustre-MDT0001 complete [ 2770.309225] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2771.573892] Key type lgssc unregistered [ 2771.712479] LNet: 115539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.716594] LNetError: 115539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.726323] LNet: Removed LNI 192.168.204.115@tcp [ 2772.046275] Key type .llcrypt unregistered [ 2772.047928] Key type ._llcrypt unregistered [ 2782.157302] Key type ._llcrypt registered [ 2782.158552] Key type .llcrypt registered [ 2782.201730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2782.548676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.557262] alg: No test for adler32 (adler32-zlib) [ 2783.414597] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2783.507668] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2785.096174] Key type lgssc registered [ 2785.629089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2788.594473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2788.602134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.797797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.273515] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2795.354384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.533234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.168393] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2799.666914] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2801.181185] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.003713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2804.156848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.568142] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2809.517583] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2814.376938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2817.572165] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2819.069030] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2819.072696] Lustre: Skipped 2 previous similar messages [ 2820.651636] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2820.655614] Lustre: Skipped 2 previous similar messages [ 2824.705740] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2824.720151] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2828.200912] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2828.206313] Lustre: Skipped 2 previous similar messages [ 2832.394331] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp [ 2834.051487] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.15/24@tcp in log lustre-MDT0000 [ 2834.055628] Lustre: Skipped 2 previous similar messages [ 2840.061769] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 2840.064524] Lustre: Skipped 1 previous similar message [ 2844.631785] Lustre: server umount lustre-OST0000 complete [ 2846.689469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2846.692541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.700688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.273745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.274347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.394285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2855.396697] Lustre: Skipped 2 previous similar messages [ 2855.425058] Lustre: server umount lustre-MDT0000 complete [ 2856.834118] LustreError: 116896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030417 with bad export cookie 11973259533645489348 [ 2856.835094] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.840153] LustreError: 116896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2867.239335] Lustre: server umount lustre-MDT0001 complete [ 2871.411096] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2872.597849] Key type lgssc unregistered [ 2872.745648] LNet: 121780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.749625] LNetError: 121780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.760444] LNet: Removed LNI 192.168.204.115@tcp [ 2873.146050] Key type .llcrypt unregistered [ 2873.147290] Key type ._llcrypt unregistered [ 2884.762053] Key type ._llcrypt registered [ 2884.763853] Key type .llcrypt registered [ 2884.807693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2885.272639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2885.292437] alg: No test for adler32 (adler32-zlib) [ 2886.161441] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2886.258964] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2887.856162] Key type lgssc registered [ 2888.316744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2891.449588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2891.460110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.661765] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2892.673352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.674471] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.15/24@tcp [ 2894.361521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2898.554631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.727863] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2898.738686] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 2898.770571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2900.572477] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2903.293978] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2904.700320] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2907.449323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.654385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.190899] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2912.984412] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2917.931401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2922.981230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.981991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2922.983290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.986200] Lustre: Skipped 1 previous similar message [ 2924.691620] Lustre: server umount lustre-OST0000 complete [ 2928.097758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.098790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.103372] Lustre: Skipped 1 previous similar message [ 2928.106017] Lustre: Skipped 1 previous similar message [ 2928.224648] Lustre: server umount lustre-MDT0000 complete [ 2929.674421] LustreError: 125649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030490 with bad export cookie 15845624341569021114 [ 2929.678818] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.680557] LustreError: 125649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2929.805844] Lustre: server umount lustre-MDT0001 complete [ 2933.991716] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2935.244933] Key type lgssc unregistered [ 2935.391945] LNet: 126670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2935.396795] LNetError: 126670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2935.409909] LNet: Removed LNI 192.168.204.115@tcp [ 2935.758476] Key type .llcrypt unregistered [ 2935.760259] Key type ._llcrypt unregistered [ 2946.634806] Key type ._llcrypt registered [ 2946.636139] Key type .llcrypt registered [ 2946.682759] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2953.520246] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2953.906404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.966908] alg: No test for adler32 (adler32-zlib) [ 2954.842565] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 2954.939966] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2956.536196] Key type lgssc registered [ 2957.090095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2972.023699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2972.034611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.139423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2973.154990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2973.195422] Lustre: lustre-MDT0000: new disk, initializing [ 2973.241794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.249418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2974.901826] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2980.606676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.657942] Lustre: 130017:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2980.698172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2980.701267] Lustre: Skipped 1 previous similar message [ 2980.744547] Lustre: lustre-MDT0001: new disk, initializing [ 2980.784884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.805232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2980.811392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2982.423268] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2986.506633] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.093823] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2990.969914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.101842] Lustre: lustre-OST0000: new disk, initializing [ 2991.103583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2991.107186] Lustre: 131855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2991.151896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.822690] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2997.255452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2997.260331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2997.296089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2998.193422] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.764395] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2999.863931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3001.970752] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3002.083534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3007.461308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.461844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3007.462317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.466768] Lustre: Skipped 1 previous similar message [ 3007.471423] LustreError: Skipped 1 previous similar message [ 3012.580297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3012.585635] Lustre: Skipped 2 previous similar messages [ 3017.184126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3017.252246] Lustre: server umount lustre-OST0000 complete [ 3021.793783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.799641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.806199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.819286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.071166] Lustre: server umount lustre-MDT0000 complete [ 3026.666738] LustreError: 130010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030587 with bad export cookie 14825250900124010668 [ 3026.670526] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.673893] LustreError: 130010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3026.838188] Lustre: server umount lustre-MDT0001 complete [ 3030.592812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.851436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.503333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3036.129972] LustreError: 133970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.139285] LustreError: 133970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3036.828804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.001502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.673238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3041.664126] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3043.224586] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3046.088787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3046.241385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.648386] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3051.295838] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.075721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3056.253792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3057.938971] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3058.042936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3062.120839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.210489] Lustre: lustre-OST0001: new disk, initializing [ 3062.212861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3062.216558] Lustre: 137129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3062.265723] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.904475] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3067.411067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3067.417160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3067.453648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3069.376100] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3069.454963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3071.097773] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3071.204644] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3072.636560] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e7247984000.ost_server_uuid 50 [ 3073.240199] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e7247984000.ost_server_uuid in FULL state after 0 sec [ 3077.604470] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.607626] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.607717] LustreError: Skipped 1 previous similar message [ 3077.615741] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3077.619367] Lustre: Skipped 3 previous similar messages [ 3081.485751] Lustre: server umount lustre-OST0001 complete [ 3082.724845] LustreError: 135658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.740176] LustreError: 135658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3083.234392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.240142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.246429] Lustre: Skipped 1 previous similar message [ 3083.251498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.846371] LustreError: 135659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.855673] LustreError: 135659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3089.263542] Lustre: server umount lustre-OST0000 complete [ 3092.962096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.962680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.973611] Lustre: Skipped 2 previous similar messages [ 3092.976657] Lustre: Skipped 3 previous similar messages [ 3097.346549] Lustre: server umount lustre-MDT0000 complete [ 3098.083890] LustreError: 133969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.091685] LustreError: 133969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3099.141176] LustreError: 137128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030659 with bad export cookie 14825250900124011550 [ 3099.141453] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3099.149990] LustreError: 137128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3099.319149] Lustre: server umount lustre-MDT0001 complete [ 3108.014783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3109.508889] Key type lgssc unregistered [ 3109.690615] LNet: 140009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3109.696182] LNetError: 140009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3109.712747] LNet: Removed LNI 192.168.204.115@tcp [ 3110.204478] Key type .llcrypt unregistered [ 3110.206492] Key type ._llcrypt unregistered [ 3129.154593] Key type ._llcrypt registered [ 3129.155971] Key type .llcrypt registered [ 3129.213826] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3136.667761] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3137.229510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3137.279673] alg: No test for adler32 (adler32-zlib) [ 3138.210248] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3138.338228] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3139.968183] Key type lgssc registered [ 3140.440747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3156.028112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3156.036906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.170533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3157.190620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3157.228739] Lustre: lustre-MDT0000: new disk, initializing [ 3157.263085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.273918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3159.014696] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3164.583875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.638266] Lustre: 144103:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3164.669262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3164.671477] Lustre: Skipped 1 previous similar message [ 3164.715402] Lustre: lustre-MDT0001: new disk, initializing [ 3164.763090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.778379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3164.782317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3166.353739] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3170.335223] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.686937] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3174.324754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.444888] Lustre: lustre-OST0000: new disk, initializing [ 3174.446968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3174.450631] Lustre: 145942:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3174.482925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.721203] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3180.616929] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3181.056318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3181.061486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3181.104629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3188.042530] Lustre: Failing over lustre-OST0000 [ 3188.122467] Lustre: server umount lustre-OST0000 complete [ 3201.130149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.237954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.248875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.255857] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3202.499934] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3203.276101] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3205.131719] Lustre: Failing over lustre-MDT0000 [ 3205.249952] Lustre: server umount lustre-MDT0000 complete [ 3205.602342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.606238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.611435] Lustre: Skipped 1 previous similar message [ 3206.628168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.634859] Lustre: Skipped 1 previous similar message [ 3207.649853] Lustre: 142329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781030752/real 1781030752] req@0000000092170e49 x1867546046263168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781030768 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3211.745756] LustreError: 144117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.753725] LustreError: 144117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3212.730782] LustreError: 146589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.768239] Lustre: 142329:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781030757/real 1781030757] req@00000000c45eee72 x1867546046264832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781030773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3216.865569] LustreError: 144938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.873423] LustreError: 144938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3218.768301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.865153] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3218.871283] Lustre: 144108:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3218.976876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3219.011955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.020500] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3219.020855] Lustre: lustre-OST0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted. [ 3219.047504] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3220.603955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3222.861108] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3224.045078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3224.050975] Lustre: Skipped 1 previous similar message [ 3224.058634] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3224.092228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3229.158893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3229.159387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.160065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3229.162148] LustreError: Skipped 1 previous similar message [ 3229.167880] Lustre: Skipped 1 previous similar message [ 3231.474261] Lustre: server umount lustre-OST0000 complete [ 3234.274281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.274980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3234.275563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.275569] Lustre: Skipped 1 previous similar message [ 3234.280655] Lustre: Skipped 1 previous similar message [ 3239.071886] Lustre: server umount lustre-MDT0000 complete [ 3239.393094] LustreError: 144117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.400494] LustreError: 144117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3240.537539] LustreError: 144096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030801 with bad export cookie 14927164690393239136 [ 3240.541794] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.544609] LustreError: 144096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.675802] Lustre: server umount lustre-MDT0001 complete [ 3244.585845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3245.845770] Key type lgssc unregistered [ 3246.003421] LNet: 149570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3246.007800] LNetError: 149570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3246.016034] LNet: Removed LNI 192.168.204.115@tcp [ 3246.386120] Key type .llcrypt unregistered [ 3246.387994] Key type ._llcrypt unregistered [ 3268.061121] Key type ._llcrypt registered [ 3268.063442] Key type .llcrypt registered [ 3268.118501] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3268.650029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3268.662611] alg: No test for adler32 (adler32-zlib) [ 3269.559448] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3269.697362] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3271.320175] Key type lgssc registered [ 3271.869539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3275.196160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3275.205075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.363414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.722786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3281.431775] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3283.538770] LustreError: 151100:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3283.543372] LustreError: 151100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3283.548296] LustreError: 151100:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3283.681384] Lustre: server umount lustre-MDT0000 complete [ 3289.432708] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3293.111950] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3308.202912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.306203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3308.324262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3308.362618] Lustre: lustre-MDT0000: new disk, initializing [ 3308.398726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.407956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3310.066084] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3315.273057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.340918] Lustre: 154300:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3315.376203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3315.378660] Lustre: Skipped 1 previous similar message [ 3315.417902] Lustre: lustre-MDT0001: new disk, initializing [ 3315.470361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.490732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3315.495135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3316.999915] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3321.074424] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3322.582219] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3325.543541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.682896] Lustre: lustre-OST0000: new disk, initializing [ 3325.686491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3325.691419] Lustre: 156134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3325.741285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.001947] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3331.962810] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3335.167173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3335.171653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3335.211793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3340.614490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3341.294105] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:48:20 (1781030900) [ 3432.416885] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.417201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.422514] Lustre: Skipped 2 previous similar messages [ 3432.426646] Lustre: Skipped 2 previous similar messages [ 3434.726184] Lustre: server umount lustre-MDT0000 complete [ 3436.023253] LustreError: 154292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781030996 with bad export cookie 10995711528604948007 [ 3436.024042] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.027970] LustreError: 154292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3437.536438] LustreError: 154306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.536503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.543599] LustreError: 154306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3437.548505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3441.569721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3441.569721] LustreError: 154305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.665846] LustreError: 154305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.665924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3450.336118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3450.420425] Lustre: server umount lustre-MDT0001 complete [ 3466.208584] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3466.266385] Lustre: server umount lustre-OST0000 complete [ 3469.619664] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3473.220285] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3485.639471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.727673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3485.740671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3485.775641] Lustre: lustre-MDT0000: new disk, initializing [ 3485.800947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.807992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3487.064474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3491.577213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.626993] Lustre: 160668:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3491.677654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3491.678914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3491.683083] Lustre: Skipped 1 previous similar message [ 3491.724142] Lustre: lustre-MDT0001: new disk, initializing [ 3491.770883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3491.774900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3493.163123] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3496.809176] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3498.059825] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3500.610138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3500.720360] Lustre: lustre-OST0000: new disk, initializing [ 3500.722960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3500.726573] Lustre: 162505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3500.759651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.762630] Lustre: Skipped 1 previous similar message [ 3502.968624] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3506.727755] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3507.718617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3507.722900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3507.754856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3508.492920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3508.584372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3510.102371] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3510.181365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3512.804249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.804523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.807781] LustreError: Skipped 1 previous similar message [ 3512.810167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.814375] Lustre: Skipped 1 previous similar message [ 3517.027464] Lustre: server umount lustre-OST0000 complete [ 3522.529047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3522.532337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.641673] Lustre: server umount lustre-MDT0000 complete [ 3526.084195] LustreError: 160660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031086 with bad export cookie 10995711528605106816 [ 3526.087677] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3526.089720] LustreError: 160660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3526.228387] Lustre: server umount lustre-MDT0001 complete [ 3529.349744] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:51:29 (1781031089) [ 3534.895289] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3538.029150] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3550.850241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.948436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3550.962871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3551.005597] Lustre: lustre-MDT0000: new disk, initializing [ 3551.038674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.050907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3552.504776] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3557.135059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.176553] Lustre: 167123:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3557.272016] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3558.538504] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3561.916456] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3563.058242] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3565.332633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.440309] Lustre: 168968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3567.376150] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3567.669874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3567.700799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3570.765372] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3572.706157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3572.708055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.710223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.713280] Lustre: Skipped 2 previous similar messages [ 3572.717392] Lustre: Skipped 5 previous similar messages [ 3578.770154] Lustre: server umount lustre-OST0000 complete [ 3582.946499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.947075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.953853] Lustre: Skipped 1 previous similar message [ 3582.956495] LustreError: Skipped 1 previous similar message [ 3586.146556] Lustre: server umount lustre-MDT0000 complete [ 3587.353089] LustreError: 167114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031148 with bad export cookie 10995711528605107705 [ 3587.356290] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.357536] LustreError: 167114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3587.475474] Lustre: server umount lustre-MDT0001 complete [ 3590.142433] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:52:29 (1781031149) [ 3595.222201] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3598.401420] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3610.626217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.718047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3610.731661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3610.734295] Lustre: Skipped 3 previous similar messages [ 3610.771369] Lustre: lustre-MDT0000: new disk, initializing [ 3610.773302] Lustre: Skipped 2 previous similar messages [ 3610.808046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.810772] Lustre: Skipped 2 previous similar messages [ 3610.819528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3610.824123] Lustre: Skipped 2 previous similar messages [ 3612.220996] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3616.989556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.039837] Lustre: 173182:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3617.081859] LustreError: 173195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.164352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3618.592190] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3622.255301] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3623.531794] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3625.816201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3625.939833] Lustre: 175029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3627.933402] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3629.206088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3631.502497] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3634.658778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.659827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.660481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.660484] Lustre: Skipped 4 previous similar messages [ 3634.663557] Lustre: Skipped 1 previous similar message [ 3634.666709] LustreError: Skipped 1 previous similar message [ 3639.505311] Lustre: server umount lustre-OST0000 complete [ 3642.849645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3655.136154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3655.207617] Lustre: server umount lustre-MDT0000 complete [ 3656.462233] LustreError: 174014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031217 with bad export cookie 10995711528605108776 [ 3656.465184] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.467089] LustreError: 174014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.597366] Lustre: server umount lustre-MDT0001 complete [ 3660.405103] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3661.511990] Key type lgssc unregistered [ 3661.653443] LNet: 176803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.656891] LNetError: 176803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3661.669965] LNet: Removed LNI 192.168.204.115@tcp [ 3662.012507] Key type .llcrypt unregistered [ 3662.013886] Key type ._llcrypt unregistered [ 3671.829831] Key type ._llcrypt registered [ 3671.831319] Key type .llcrypt registered [ 3671.872254] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3672.287151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.301829] alg: No test for adler32 (adler32-zlib) [ 3673.165567] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3673.254070] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3674.840138] Key type lgssc registered [ 3675.227135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3678.038946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3678.047427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.211343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.548171] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3683.832439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.978097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3685.357081] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3687.663751] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.879438] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3691.071272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.180853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.069560] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3695.189504] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3703.466568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3719.139161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.140542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.146575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.149246] Lustre: Skipped 1 previous similar message [ 3724.257830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.555828] Lustre: server umount lustre-OST0000 complete [ 3729.377622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.378319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.381148] Lustre: Skipped 2 previous similar messages [ 3729.382753] Lustre: Skipped 1 previous similar message [ 3734.497034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3734.499613] Lustre: Skipped 2 previous similar messages [ 3739.617245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.620467] Lustre: Skipped 1 previous similar message [ 3740.128132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3740.198791] Lustre: server umount lustre-MDT0000 complete [ 3741.506638] LustreError: 178160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031302 with bad export cookie 15752254483486232393 [ 3741.509637] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.511778] LustreError: 178160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3741.630863] Lustre: server umount lustre-MDT0001 complete [ 3745.136278] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3746.175884] Key type lgssc unregistered [ 3746.306352] LNet: 181520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3746.309131] LNetError: 181520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.319297] LNet: Removed LNI 192.168.204.115@tcp [ 3746.610650] Key type .llcrypt unregistered [ 3746.611767] Key type ._llcrypt unregistered [ 3755.569858] Key type ._llcrypt registered [ 3755.570878] Key type .llcrypt registered [ 3755.607168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3755.975883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3756.032436] alg: No test for adler32 (adler32-zlib) [ 3756.908978] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 3757.000123] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3758.592176] Key type lgssc registered [ 3759.052619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3761.900191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3761.906992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.073299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.482707] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3767.957418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.086887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.579456] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3771.955863] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3773.242137] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3775.753408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.885622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.873640] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3780.211561] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3788.201362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3793.379509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3793.380224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.381728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.382434] LustreError: Skipped 1 previous similar message [ 3793.387301] Lustre: Skipped 1 previous similar message [ 3795.907904] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 3795.910481] Lustre: Skipped 1 previous similar message [ 3798.498787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3798.503295] Lustre: Skipped 1 previous similar message [ 3799.254612] Lustre: server umount lustre-OST0000 complete [ 3801.186657] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e72480ca800.ost_server_uuid 50 [ 3801.759652] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3811.732157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.732700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.736270] Lustre: Skipped 1 previous similar message [ 3816.929658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3816.932256] Lustre: Skipped 2 previous similar messages [ 3817.883335] Lustre: server umount lustre-MDT0000 complete [ 3819.164146] LustreError: 182877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031379 with bad export cookie 16152635789949509717 [ 3819.167599] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3819.168394] LustreError: 182877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3819.287790] Lustre: server umount lustre-MDT0001 complete [ 3822.164185] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:56:21 (1781031381) [ 3824.482949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.627250] LustreError: 186518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.635620] LustreError: 186518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3824.654576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3825.998404] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3829.230714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.359949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3830.705978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3832.934172] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3834.171907] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3836.512063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.621492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3840.910767] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3843.277687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.367752] Lustre: lustre-OST0001: new disk, initializing [ 3843.369879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3843.372634] Lustre: 189161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3843.406128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3843.408426] Lustre: Skipped 1 previous similar message [ 3845.360539] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3845.505453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3845.509076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3845.542063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3848.840203] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3849.641116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3850.209168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.211518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.215387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.819424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.821346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.802981] Lustre: server umount lustre-OST0000 complete [ 3857.773823] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3857.859962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3859.089444] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3859.159048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3859.387218] LustreError: 188205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.508020] LustreError: 189527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.514142] LustreError: 189527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3870.178942] LustreError: 188205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.179788] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3870.180738] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3870.180746] Lustre: Skipped 3 previous similar messages [ 3870.185961] LustreError: 188205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3870.190668] Lustre: Skipped 1 previous similar message [ 3875.279374] Lustre: server umount lustre-OST0001 complete [ 3880.418668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.418712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.423841] Lustre: Skipped 1 previous similar message [ 3882.716116] Lustre: server umount lustre-MDT0000 complete [ 3883.943387] LustreError: 186499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031444 with bad export cookie 16152635789949510774 [ 3883.944858] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.950752] LustreError: 186499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3884.087495] Lustre: server umount lustre-MDT0001 complete [ 3891.525192] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:57:31 (1781031451) [ 3893.749440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.834548] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3893.847564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3893.923725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.246938] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3898.525042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.562270] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3898.574921] Lustre: 192485:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3899.903196] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3901.986168] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3903.202203] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3905.435626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.498107] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3907.152245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3907.434443] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3909.538250] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3911.723864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.750781] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3913.662376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3915.814660] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3917.283363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.287307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.290882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.293247] Lustre: Skipped 1 previous similar message [ 3917.294913] Lustre: Skipped 3 previous similar messages [ 3917.338336] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3922.747759] Lustre: server umount lustre-OST0000 complete [ 3926.459820] LustreError: 194201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.762888] LustreError: 194199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.764218] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.768328] LustreError: 194199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3937.770184] LustreError: Skipped 2 previous similar messages [ 3939.472949] Lustre: server umount lustre-OST0001 complete [ 3946.840399] Lustre: server umount lustre-MDT0000 complete [ 3948.139087] LustreError: 192475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031508 with bad export cookie 16152635789949512160 [ 3948.141701] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.143396] LustreError: 192475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3948.265846] Lustre: server umount lustre-MDT0001 complete [ 3955.966139] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:58:35 (1781031515) [ 3961.336253] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3964.491638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3976.047061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.133265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3976.145211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3976.180644] Lustre: lustre-MDT0000: new disk, initializing [ 3976.202873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.205202] Lustre: Skipped 3 previous similar messages [ 3976.211353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3977.547541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3982.198079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.244244] Lustre: 200244:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3982.270588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3982.272560] Lustre: Skipped 1 previous similar message [ 3982.305827] Lustre: lustre-MDT0001: new disk, initializing [ 3982.351855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3982.355421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3983.663326] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3987.007195] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3988.088713] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3990.161559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3990.249320] Lustre: lustre-OST0000: new disk, initializing [ 3990.251100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3990.253168] Lustre: 202082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3992.019222] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3994.615190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3994.619569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3994.648518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3995.352695] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3996.653984] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3996.729107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3997.949561] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3998.012187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3999.715478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.717252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.718625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.718630] Lustre: Skipped 8 previous similar messages [ 3999.719817] LustreError: Skipped 1 previous similar message [ 3999.724779] Lustre: Skipped 5 previous similar messages [ 4000.714451] Lustre: server umount lustre-OST0000 complete [ 4008.160833] Lustre: server umount lustre-MDT0000 complete [ 4009.393649] LustreError: 200238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031570 with bad export cookie 16152635789949513903 [ 4009.397233] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.397811] LustreError: 200238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4009.526280] Lustre: server umount lustre-MDT0001 complete [ 4012.353073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.516712] LustreError: 204196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.524839] LustreError: 204196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4013.762366] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4016.864475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.242984] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4020.292969] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4021.461616] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4023.728406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4025.721293] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4027.788039] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4029.105441] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.363696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 4040.686154] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4040.761284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.208867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.211388] LustreError: Skipped 1 previous similar message [ 4047.498034] Lustre: server umount lustre-OST0000 complete [ 4049.458873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4050.827264] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4074.438587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.543124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.545806] Lustre: Skipped 5 previous similar messages [ 4076.409481] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4078.455363] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.971140] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4086.021578] Lustre: server umount lustre-OST0000 complete [ 4088.800827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.804874] Lustre: Skipped 4 previous similar messages [ 4093.395366] Lustre: server umount lustre-MDT0000 complete [ 4094.434059] LustreError: 204197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.439704] LustreError: 204197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4094.650197] LustreError: 208739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031655 with bad export cookie 16152635789949514771 [ 4094.653137] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.655172] LustreError: 208739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4094.768165] Lustre: server umount lustre-MDT0001 complete [ 4097.719077] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:00:57 (1781031657) [ 4100.061355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.731978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4105.236909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.832215] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4109.080318] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4110.334941] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4112.680202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4114.766517] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4116.978528] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4118.441331] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4126.678625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4128.018783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4128.093787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4130.348326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.420885] Lustre: lustre-OST0001: new disk, initializing [ 4130.423376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4130.425963] Lustre: 213104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4131.147358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4131.151699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4131.185971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4132.320327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4135.769160] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4137.136137] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4137.219054] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4138.592925] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4138.659486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4141.542578] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.545220] LustreError: Skipped 3 previous similar messages [ 4141.546897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4141.549297] Lustre: Skipped 12 previous similar messages [ 4145.402255] Lustre: server umount lustre-OST0001 complete [ 4147.256885] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4147.330737] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4148.600294] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4148.664795] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4158.907872] LustreError: 211598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.913391] LustreError: 211598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4172.124614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.990688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4176.051137] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4176.575416] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4183.291740] Lustre: server umount lustre-OST0001 complete [ 4201.427831] Lustre: server umount lustre-OST0000 complete [ 4208.928978] Lustre: server umount lustre-MDT0000 complete [ 4210.223353] LustreError: 209877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031770 with bad export cookie 16152635789949515702 [ 4210.226793] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4210.227623] LustreError: 209877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4216.436726] Lustre: server umount lustre-MDT0001 complete [ 4223.877789] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:03:03 (1781031783) [ 4226.064221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.161221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4226.173148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4226.248361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.251827] Lustre: Skipped 5 previous similar messages [ 4227.513063] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4230.677719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.709467] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4230.720884] Lustre: 218133:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4232.061171] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4234.125038] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4235.257292] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4237.393402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4237.450775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4239.301434] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4241.485176] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4245.182258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.120872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4249.219644] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4251.762689] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4253.049966] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4253.126309] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4254.446479] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4254.539738] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4255.759537] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e724733b800.ost_server_uuid 50 [ 4256.232338] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e724733b800.ost_server_uuid in FULL state after 0 sec [ 4256.543316] Lustre: Permanently deactivating lustre-OST0001 [ 4256.547883] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4257.308969] Lustre: Permanently reactivating lustre-OST0001 [ 4261.346690] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.350601] Lustre: Skipped 8 previous similar messages [ 4264.317741] Lustre: server umount lustre-OST0001 complete [ 4271.698673] Lustre: server umount lustre-OST0000 complete [ 4278.997854] Lustre: server umount lustre-MDT0000 complete [ 4280.166429] LustreError: 218906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031840 with bad export cookie 16152635789949516962 [ 4280.169508] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.170555] LustreError: 218906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4280.292265] Lustre: server umount lustre-MDT0001 complete [ 4287.699812] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:04:07 (1781031847) [ 4290.126350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.212780] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4290.226220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4290.230789] Lustre: Skipped 5 previous similar messages [ 4291.599994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4294.727464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.762673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4294.764543] Lustre: Skipped 1 previous similar message [ 4294.773217] Lustre: 224425:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4296.142060] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4298.219279] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4299.437752] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4301.665439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.744658] Lustre: Permanently deactivating lustre-OST0000 [ 4301.749028] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4303.346628] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4303.594211] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4305.708988] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4307.947058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.399971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4309.825241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4311.942171] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4312.541458] Lustre: Permanently reactivating lustre-OST0000 [ 4317.686684] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4317.689927] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.693835] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4317.719569] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4317.728691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4323.814419] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4323.816317] LustreError: Skipped 5 previous similar messages [ 4326.461892] Lustre: server umount lustre-OST0001 complete [ 4329.958288] LustreError: 227860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.963316] LustreError: 227860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4333.773363] Lustre: server umount lustre-OST0000 complete [ 4337.181415] Lustre: server umount lustre-MDT0000 complete [ 4338.406651] LustreError: 225196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031899 with bad export cookie 16152635789949518614 [ 4338.410017] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4338.410141] LustreError: 225196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4338.531748] Lustre: server umount lustre-MDT0001 complete [ 4342.177885] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4343.210918] Key type lgssc unregistered [ 4343.339371] LNet: 228997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4343.342316] LNetError: 228997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4343.351286] LNet: Removed LNI 192.168.204.115@tcp [ 4343.635910] Key type .llcrypt unregistered [ 4343.637078] Key type ._llcrypt unregistered [ 4352.742846] Key type ._llcrypt registered [ 4352.744098] Key type .llcrypt registered [ 4352.780976] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4353.339930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4353.351144] alg: No test for adler32 (adler32-zlib) [ 4354.222176] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4354.311161] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4355.896181] Key type lgssc registered [ 4356.270882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4359.751619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4359.757335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.922161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.286774] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4365.610985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.655837] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4365.665698] Lustre: 230412:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4365.669838] Lustre: Permanently deactivating lustre-MDT0001 [ 4365.674812] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4365.792160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4365.965910] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4365.968850] LustreError: 230453:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4365.973260] LustreError: 230453:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4365.977353] LustreError: 230453:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 4367.148988] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4369.441527] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4370.625412] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4372.801290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4372.916910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.658487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4375.973495] LustreError: 232132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.976633] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4375.978994] LustreError: 232132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4376.634824] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4378.740144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.809494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4380.533238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4382.518058] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4384.236803] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4385.260963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4385.261315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4388.184021] Lustre: Permanently reactivating lustre-MDT0001 [ 4388.190814] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4388.193671] Lustre: Skipped 2 previous similar messages [ 4390.399909] Lustre: lustre-MDT0001: Client 116eb584-d3ca-439f-8ee7-4671ca8ec408 (at 192.168.204.15@tcp) reconnecting [ 4392.299889] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4392.764643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4394.390863] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4394.461953] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4395.746669] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4395.831761] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4396.603807] Lustre: Permanently deactivating lustre-MDT0001 [ 4396.608970] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4396.611054] Lustre: Skipped 2 previous similar messages [ 4397.439888] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4400.611314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.612738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.615675] Lustre: Skipped 2 previous similar messages [ 4400.617933] Lustre: Skipped 1 previous similar message [ 4405.729205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.731015] Lustre: Skipped 3 previous similar messages [ 4406.301287] Lustre: server umount lustre-MDT0000 complete [ 4407.528250] LustreError: 230404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781031968 with bad export cookie 14415132791322674856 [ 4407.530676] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.531157] LustreError: 230404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4407.651433] Lustre: server umount lustre-MDT0001 complete [ 4414.976104] Lustre: server umount lustre-OST0000 complete [ 4418.385548] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:06:18 (1781031978) [ 4421.757226] Lustre: server umount lustre-OST0001 complete [ 4423.609086] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4426.214599] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4436.980946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.077541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4437.080834] Lustre: Skipped 2 previous similar messages [ 4437.089080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4437.127949] Lustre: lustre-MDT0000: new disk, initializing [ 4437.150171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.155961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4438.394225] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4442.662341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.699487] Lustre: 238241:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4442.727138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4442.729735] Lustre: Skipped 1 previous similar message [ 4442.772484] Lustre: lustre-MDT0001: new disk, initializing [ 4442.822913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4442.826250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4444.097159] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4447.428706] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4448.557734] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4450.658100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4450.753404] Lustre: lustre-OST0000: new disk, initializing [ 4450.754839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4450.757121] Lustre: 240078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4450.784707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.786497] Lustre: Skipped 1 previous similar message [ 4452.606180] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4456.092736] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4456.439610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4456.443328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4456.473815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4462.022732] LustreError: 238277:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4466.168092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.230944] Lustre: lustre-OST0001: new disk, initializing [ 4466.232568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4466.234832] Lustre: 241131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4468.127248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4471.577809] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4472.112107] LustreError: 238277:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4472.119877] LustreError: 241727:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4473.119459] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4473.341345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4473.344376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4473.372508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4482.208124] LustreError: 241727:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4483.556751] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4483.557889] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.559706] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4483.563793] Lustre: Skipped 2 previous similar messages [ 4487.619299] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 4487.621750] Lustre: Skipped 1 previous similar message [ 4492.731666] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 4492.734328] Lustre: Skipped 2 previous similar messages [ 4497.376099] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4497.406491] Lustre: server umount lustre-OST0001 complete [ 4497.851285] LustreError: 240062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.916242] LustreError: 240064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.921855] LustreError: 240064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4504.036409] LustreError: 240732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.042361] LustreError: 240732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4509.156322] LustreError: 242039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.161732] LustreError: 242039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4512.737498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4512.737536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.740453] LustreError: Skipped 2 previous similar messages [ 4512.745852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.748966] Lustre: Skipped 3 previous similar messages [ 4515.543902] Lustre: server umount lustre-OST0000 complete [ 4519.393572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.396933] Lustre: Skipped 2 previous similar messages [ 4523.169580] Lustre: server umount lustre-MDT0000 complete [ 4524.398132] LustreError: 241130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032085 with bad export cookie 14415132791322677670 [ 4524.401353] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.403986] LustreError: 241130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4530.616659] Lustre: server umount lustre-MDT0001 complete [ 4534.231748] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4535.239974] Key type lgssc unregistered [ 4535.370321] LNet: 243082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4535.373059] LNetError: 243082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4535.381272] LNet: Removed LNI 192.168.204.115@tcp [ 4535.661106] Key type .llcrypt unregistered [ 4535.662454] Key type ._llcrypt unregistered [ 4549.699838] Key type ._llcrypt registered [ 4549.700924] Key type .llcrypt registered [ 4549.734981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4550.157289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4550.168221] alg: No test for adler32 (adler32-zlib) [ 4551.016460] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4551.091992] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4552.664125] Key type lgssc registered [ 4552.979226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4555.309429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4555.314893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.397803] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4556.404758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4556.463704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.752620] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4560.842865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.875068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4560.886629] Lustre: 245249:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.982694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4562.197616] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4564.282172] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4565.471467] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4567.623967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.689472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4567.746993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.470864] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4571.079106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4571.465502] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4612.066837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4612.067040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.069377] LustreError: Skipped 1 previous similar message [ 4612.074643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4612.077641] Lustre: Skipped 1 previous similar message [ 4615.720330] Lustre: server umount lustre-OST0000 complete [ 4617.155701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4621.098471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4621.212795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.944258] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4625.048847] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4631.460515] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.463661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.463725] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4631.468469] Lustre: Skipped 1 previous similar message [ 4660.196978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.197713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.202051] Lustre: Skipped 2 previous similar messages [ 4664.284329] Lustre: server umount lustre-OST0000 complete [ 4668.384858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4668.387270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.391665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.393225] Lustre: Skipped 1 previous similar message [ 4670.432974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.433475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.705880] Lustre: server umount lustre-MDT0000 complete [ 4672.922072] LustreError: 245239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032233 with bad export cookie 4304365460862845378 [ 4672.925130] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.925451] LustreError: 245239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4673.039485] Lustre: server umount lustre-MDT0001 complete [ 4676.509659] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4677.532851] Key type lgssc unregistered [ 4677.663380] LNet: 250385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4677.665968] LNetError: 250385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4677.674395] LNet: Removed LNI 192.168.204.115@tcp [ 4677.960877] Key type .llcrypt unregistered [ 4677.962246] Key type ._llcrypt unregistered [ 4686.892519] Key type ._llcrypt registered [ 4686.893683] Key type .llcrypt registered [ 4686.929626] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4687.261693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4687.300095] alg: No test for adler32 (adler32-zlib) [ 4688.164394] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4688.246653] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4689.832206] Key type lgssc registered [ 4690.244493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4692.927449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4692.934925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.103331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.559989] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4698.883874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.009063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4700.314166] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4702.465904] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4703.612743] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4705.764221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4705.866473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.760281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4709.752234] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4719.207609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4729.827584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4729.829168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.832091] LustreError: Skipped 1 previous similar message [ 4729.837429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4729.840111] Lustre: Skipped 1 previous similar message [ 4733.522206] Lustre: server umount lustre-OST0000 complete [ 4734.945594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4734.945807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.953305] Lustre: Skipped 1 previous similar message [ 4734.955480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4734.958295] Lustre: Skipped 1 previous similar message [ 4740.065204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.067202] Lustre: Skipped 1 previous similar message [ 4745.185288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4745.188221] Lustre: Skipped 1 previous similar message [ 4749.280124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.281291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4749.285877] Lustre: Skipped 1 previous similar message [ 4749.343986] Lustre: server umount lustre-MDT0000 complete [ 4750.568136] LustreError: 253474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032311 with bad export cookie 11796599515548166959 [ 4750.571673] LustreError: 253474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4750.571724] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4750.687559] Lustre: server umount lustre-MDT0001 complete [ 4754.510890] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4755.689659] Key type lgssc unregistered [ 4755.821457] LNet: 255877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4755.824408] LNetError: 255877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4755.832303] LNet: Removed LNI 192.168.204.115@tcp [ 4756.155992] Key type .llcrypt unregistered [ 4756.157161] Key type ._llcrypt unregistered [ 4762.855434] Key type ._llcrypt registered [ 4762.856494] Key type .llcrypt registered [ 4762.892519] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4763.283927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4763.295087] alg: No test for adler32 (adler32-zlib) [ 4764.136350] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4764.207324] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4765.784126] Key type lgssc registered [ 4766.153666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4768.488056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4768.493258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.629646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.832635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4773.868731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.993224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4775.282237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4777.286347] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4778.432439] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4780.543342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.645329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.282681] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4784.220322] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.008668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4804.066627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.067388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4804.070532] Lustre: Skipped 1 previous similar message [ 4804.073135] Lustre: Skipped 1 previous similar message [ 4809.185464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4809.189020] Lustre: Skipped 1 previous similar message [ 4809.616200] Lustre: server umount lustre-OST0000 complete [ 4814.305697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.306500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.309475] Lustre: Skipped 1 previous similar message [ 4814.312529] Lustre: Skipped 1 previous similar message [ 4817.048273] Lustre: server umount lustre-MDT0000 complete [ 4818.320143] LustreError: 257040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032379 with bad export cookie 9961567722667939782 [ 4818.323548] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.323781] LustreError: 257040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4818.463577] Lustre: server umount lustre-MDT0001 complete [ 4822.110994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4823.119864] Key type lgssc unregistered [ 4823.245325] LNet: 260538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4823.249120] LNetError: 260538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4823.259277] LNet: Removed LNI 192.168.204.115@tcp [ 4823.578118] Key type .llcrypt unregistered [ 4823.579588] Key type ._llcrypt unregistered [ 4832.440561] Key type ._llcrypt registered [ 4832.442258] Key type .llcrypt registered [ 4832.484350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4833.034877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4833.056865] alg: No test for adler32 (adler32-zlib) [ 4833.915732] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4834.008345] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4835.584114] Key type lgssc registered [ 4835.949103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.557523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4838.562781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.735885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.145552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4844.385505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.526075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4845.935194] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4848.084543] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4849.308368] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.545790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.662487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.610265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4855.791225] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4865.000603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4875.234069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4875.235851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.237569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4875.241142] Lustre: Skipped 1 previous similar message [ 4880.014787] Lustre: server umount lustre-OST0000 complete [ 4885.474221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.474442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4885.474770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.474776] Lustre: Skipped 1 previous similar message [ 4885.478098] Lustre: Skipped 1 previous similar message [ 4885.480166] LustreError: Skipped 1 previous similar message [ 4887.450132] Lustre: server umount lustre-MDT0000 complete [ 4888.681397] LustreError: 261896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032449 with bad export cookie 6413985520139983194 [ 4888.685275] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4888.686169] LustreError: 261896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4888.806502] Lustre: server umount lustre-MDT0001 complete [ 4892.321500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4893.403745] Key type lgssc unregistered [ 4893.543431] LNet: 266024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4893.545962] LNetError: 266024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4893.554378] LNet: Removed LNI 192.168.204.115@tcp [ 4893.897601] Key type .llcrypt unregistered [ 4893.898825] Key type ._llcrypt unregistered [ 4901.258882] Key type ._llcrypt registered [ 4901.260113] Key type .llcrypt registered [ 4901.297508] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4901.716359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4901.723493] alg: No test for adler32 (adler32-zlib) [ 4902.584776] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4902.681151] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4904.264218] Key type lgssc registered [ 4904.679614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4907.443540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4907.450171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.621119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.940903] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4913.232574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.371160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4914.764730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4916.985038] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4918.222341] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4920.472707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.597403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.441833] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4924.590561] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4933.932483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4939.237210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.237864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.242635] Lustre: Skipped 1 previous similar message [ 4939.243424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.245963] Lustre: Skipped 1 previous similar message [ 4940.822541] Lustre: server umount lustre-OST0000 complete [ 4944.217856] Lustre: server umount lustre-MDT0000 complete [ 4944.354377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4944.354636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.355670] LustreError: 267210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.355681] LustreError: 267210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4944.356610] LustreError: Skipped 1 previous similar message [ 4944.371254] Lustre: Skipped 1 previous similar message [ 4945.513700] LustreError: 267186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032506 with bad export cookie 11569402620226530334 [ 4945.517561] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.519329] LustreError: 267186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4945.649740] Lustre: server umount lustre-MDT0001 complete [ 4949.487797] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4950.604713] Key type lgssc unregistered [ 4950.732357] LNet: 270722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4950.735095] LNetError: 270722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4950.746349] LNet: Removed LNI 192.168.204.115@tcp [ 4951.045567] Key type .llcrypt unregistered [ 4951.046927] Key type ._llcrypt unregistered [ 4954.629931] Key type ._llcrypt registered [ 4954.631188] Key type .llcrypt registered [ 4954.668396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4960.031873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4966.147573] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4966.570798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4966.583618] alg: No test for adler32 (adler32-zlib) [ 4967.443545] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 4967.532168] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4969.112111] Key type lgssc registered [ 4969.495871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4981.282287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4981.288278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.382703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4982.395535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4982.431882] Lustre: lustre-MDT0000: new disk, initializing [ 4982.458463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.465414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4983.795473] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4988.214200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.263991] Lustre: 274195:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4988.291630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4988.293601] Lustre: Skipped 1 previous similar message [ 4988.335546] Lustre: lustre-MDT0001: new disk, initializing [ 4988.375390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4988.389900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4988.394595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4989.732775] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4993.195182] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4994.439280] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4996.803822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4996.909093] Lustre: lustre-OST0000: new disk, initializing [ 4996.912151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4996.916931] Lustre: 276031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4996.949862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.771210] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5002.220613] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5003.262234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5003.267093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.309060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5003.654599] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5003.752845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5005.088967] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5005.164407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5008.356080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.356110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.362840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.365025] LustreError: Skipped 1 previous similar message [ 5011.924552] Lustre: server umount lustre-OST0000 complete [ 5013.473522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.474129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.477596] Lustre: Skipped 2 previous similar messages [ 5013.479181] Lustre: Skipped 1 previous similar message [ 5018.593507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.597021] Lustre: Skipped 2 previous similar messages [ 5019.359737] Lustre: server umount lustre-MDT0000 complete [ 5020.596318] LustreError: 275021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032581 with bad export cookie 13911211004817769676 [ 5020.599945] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.600266] LustreError: 275021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5020.725456] Lustre: server umount lustre-MDT0001 complete [ 5023.589476] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:16:23 (1781032583) [ 5025.793922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.968615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.239106] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5030.379363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.527075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5031.871993] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5034.008773] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5035.199837] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5037.392186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5039.341722] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5041.533129] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5055.971438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5055.971932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.973258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5055.974402] LustreError: Skipped 1 previous similar message [ 5055.978966] Lustre: Skipped 1 previous similar message [ 5060.820813] Lustre: server umount lustre-OST0000 complete [ 5065.187400] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5065.189920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.196436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.199067] Lustre: Skipped 1 previous similar message [ 5068.321682] Lustre: server umount lustre-MDT0000 complete [ 5069.610426] LustreError: 278323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032630 with bad export cookie 13911211004817770544 [ 5069.613223] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.616405] LustreError: 278323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5069.739405] Lustre: server umount lustre-MDT0001 complete [ 5073.485195] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5074.562931] Key type lgssc unregistered [ 5074.688516] LNet: 281682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5074.690850] LNetError: 281682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5074.699389] LNet: Removed LNI 192.168.204.115@tcp [ 5074.994951] Key type .llcrypt unregistered [ 5074.996757] Key type ._llcrypt unregistered [ 5083.101639] Key type ._llcrypt registered [ 5083.102630] Key type .llcrypt registered [ 5083.135626] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5089.121196] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5089.513350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5089.550036] alg: No test for adler32 (adler32-zlib) [ 5090.402935] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5090.476956] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5092.056151] Key type lgssc registered [ 5092.393517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5106.347417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5106.355779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.447792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5107.460172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5107.508130] Lustre: lustre-MDT0000: new disk, initializing [ 5107.535411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.541637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5108.811492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5113.264634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.306465] Lustre: 285444:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5113.330537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5113.332409] Lustre: Skipped 1 previous similar message [ 5113.367665] Lustre: lustre-MDT0001: new disk, initializing [ 5113.399292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5113.410807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5113.414538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5114.728398] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5118.205056] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5119.425741] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5121.698735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5121.807259] Lustre: lustre-OST2710: new disk, initializing [ 5121.809361] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5121.813362] Lustre: 287279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5121.844755] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5123.714038] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5124.547239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5124.552633] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5124.600902] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5127.232730] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5129.633970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.710095] Lustre: lustre-OST03e8: new disk, initializing [ 5129.711819] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5129.714995] Lustre: 288278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5129.742161] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5131.657993] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5131.716975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5131.720597] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5131.757285] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5133.775039] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5136.555038] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5136.647542] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5138.044911] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5138.115826] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5139.535389] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5139.616370] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5141.006522] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5141.082204] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5144.032757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5144.035311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.039786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.107815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.109183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.112416] Lustre: Skipped 1 previous similar message [ 5148.321989] Lustre: server umount lustre-MDT0000 complete [ 5149.559188] LustreError: 288277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032710 with bad export cookie 4107253151465436116 [ 5149.562699] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5149.562877] LustreError: 288277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5149.675436] Lustre: server umount lustre-MDT0001 complete [ 5160.388918] Lustre: server umount lustre-OST2710 complete [ 5171.872977] Lustre: server umount lustre-OST03e8 complete [ 5177.842777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5181.190153] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5193.072717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.162930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5193.174400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5193.210337] Lustre: lustre-MDT0000: new disk, initializing [ 5193.233057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.239485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5194.567224] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5199.263282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.307569] Lustre: 293348:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5199.411600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5200.800069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5204.243526] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.493384] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.737365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5207.837540] Lustre: lustre-OST0000: new disk, initializing [ 5207.839861] Lustre: Skipped 1 previous similar message [ 5207.842487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5207.844782] Lustre: Skipped 2 previous similar messages [ 5207.846950] Lustre: 295189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5207.874842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.877085] Lustre: Skipped 1 previous similar message [ 5207.989463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5207.992319] Lustre: Skipped 1 previous similar message [ 5207.994133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5208.024786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5209.696562] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5213.116499] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5214.490182] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5214.569894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5215.859090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5215.930390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5218.275255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5218.275802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.277770] LustreError: Skipped 1 previous similar message [ 5218.278261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.281667] Lustre: Skipped 2 previous similar messages [ 5218.282963] Lustre: Skipped 3 previous similar messages [ 5222.676208] Lustre: server umount lustre-OST0000 complete [ 5224.928972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5224.931411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.935638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.937638] Lustre: Skipped 1 previous similar message [ 5233.632966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.635499] Lustre: Skipped 3 previous similar messages [ 5238.240154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5238.308311] Lustre: server umount lustre-MDT0000 complete [ 5239.579282] LustreError: 293340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032800 with bad export cookie 4107253151465437782 [ 5239.582656] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5239.584400] LustreError: 293340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5239.697270] Lustre: server umount lustre-MDT0001 complete [ 5242.387293] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:20:02 (1781032802) [ 5242.891277] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5243.437556] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:20:03 (1781032803) [ 5244.437470] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5253.014662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.095504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5253.102354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5253.156766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.464355] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5257.739627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.779098] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5257.792409] Lustre: 298924:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5259.217975] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5275.491384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.580734] Lustre: Denying initial registration attempt from nid 192.168.204.115@tcp, specified as failover [ 5275.583891] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5275.588456] LustreError: 300438:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5275.592841] LustreError: 300438:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5275.596180] LustreError: 300438:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5275.627622] Lustre: server umount lustre-OST0000 complete [ 5275.629252] LustreError: 300438:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5278.688940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5278.691285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.694860] Lustre: Skipped 1 previous similar message [ 5278.696649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.993964] Lustre: server umount lustre-MDT0000 complete [ 5285.262413] LustreError: 299694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032846 with bad export cookie 4107253151465438650 [ 5285.265707] LustreError: 299694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5285.266214] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.376553] Lustre: server umount lustre-MDT0001 complete [ 5288.776969] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5289.837051] Key type lgssc unregistered [ 5289.975423] LNet: 301540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5289.978306] LNetError: 301540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5289.986387] LNet: Removed LNI 192.168.204.115@tcp [ 5290.298068] Key type .llcrypt unregistered [ 5290.299736] Key type ._llcrypt unregistered [ 5294.029321] Key type ._llcrypt registered [ 5294.030350] Key type .llcrypt registered [ 5294.071190] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5294.450647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5294.492406] alg: No test for adler32 (adler32-zlib) [ 5295.354219] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5295.467701] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5297.056190] Key type lgssc registered [ 5297.451245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5321.154765] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5324.103778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5324.109876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.224343] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5325.231967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5325.306978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.701956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5329.852078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.886079] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5329.898109] Lustre: 304248:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5330.013646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5331.444924] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5333.616325] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5334.791285] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5350.051049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.123732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5350.199127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.110208] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5354.236157] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5354.964160] Lustre: server umount lustre-OST0000 complete [ 5358.067252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.068849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.070847] Lustre: Skipped 1 previous similar message [ 5358.076586] Lustre: Skipped 1 previous similar message [ 5362.389130] Lustre: server umount lustre-MDT0000 complete [ 5363.629353] LustreError: 305022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032924 with bad export cookie 8551111635904645557 [ 5363.632449] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.634860] LustreError: 305022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5363.756433] Lustre: server umount lustre-MDT0001 complete [ 5367.202674] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5368.183345] Key type lgssc unregistered [ 5368.315361] LNet: 307597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.319111] LNetError: 307597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.328316] LNet: Removed LNI 192.168.204.115@tcp [ 5368.616139] Key type .llcrypt unregistered [ 5368.617308] Key type ._llcrypt unregistered [ 5377.886740] Key type ._llcrypt registered [ 5377.887908] Key type .llcrypt registered [ 5377.924463] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5378.354813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5378.361780] alg: No test for adler32 (adler32-zlib) [ 5379.218141] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5379.315668] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5380.896137] Key type lgssc registered [ 5381.277072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5383.880832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5383.888288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.066341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.420528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5389.617475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.758958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5391.022405] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5393.174436] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5394.389367] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5398.755071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5398.873051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.679715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5402.762809] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5415.394151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5415.394832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.395678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5415.401031] Lustre: Skipped 1 previous similar message [ 5418.434746] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 5418.436747] Lustre: Skipped 2 previous similar messages [ 5420.514495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.516867] Lustre: Skipped 2 previous similar messages [ 5421.340112] Lustre: server umount lustre-MDT0000 complete [ 5422.586490] LustreError: 310685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781032983 with bad export cookie 7073657052594259282 [ 5422.589525] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.592754] LustreError: 310685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5423.546462] LustreError: 308970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.554270] LustreError: 308970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5423.555360] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5425.633030] LustreError: 310371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.633234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.642613] Lustre: Skipped 1 previous similar message [ 5427.680822] LustreError: 308970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.681045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5427.690130] Lustre: Skipped 1 previous similar message [ 5428.773936] Lustre: server umount lustre-MDT0001 complete [ 5430.884744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5433.675042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.160606] LustreError: 308284:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b2b67859 x1867548396117504/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5448.270178] LustreError: 310672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.277181] LustreError: 310672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5448.295769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.314468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5449.551426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5452.387455] LustreError: 312388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.393307] LustreError: 312388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5452.732376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.871817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.261523] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5456.428282] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5457.896552] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5458.917472] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5460.080676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5461.230363] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5461.760725] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5463.009407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5463.012326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.016580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5463.018986] Lustre: Skipped 1 previous similar message [ 5477.344159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5477.407729] Lustre: server umount lustre-OST0000 complete [ 5479.394845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.398751] Lustre: Skipped 1 previous similar message [ 5479.401053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.404015] Lustre: Skipped 8 previous similar messages [ 5493.216138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5493.277869] Lustre: server umount lustre-MDT0000 complete [ 5494.480261] LustreError: 308955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033055 with bad export cookie 7073657052594266436 [ 5494.483833] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5494.484862] LustreError: 308955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5494.604925] Lustre: server umount lustre-MDT0001 complete [ 5498.407637] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5499.400736] Key type lgssc unregistered [ 5499.525368] LNet: 314914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5499.528173] LNetError: 314914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5499.536321] LNet: Removed LNI 192.168.204.115@tcp [ 5499.848907] Key type .llcrypt unregistered [ 5499.850668] Key type ._llcrypt unregistered [ 5508.836151] Key type ._llcrypt registered [ 5508.837775] Key type .llcrypt registered [ 5508.876568] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5509.164547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5509.170315] alg: No test for adler32 (adler32-zlib) [ 5510.017462] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5510.104459] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5511.688112] Key type lgssc registered [ 5512.070715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5514.668749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5514.677558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.854583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.225688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5520.354761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.480612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.806353] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5524.043085] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5525.314203] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5530.403177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.404080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.408975] Lustre: Skipped 1 previous similar message [ 5532.441839] Lustre: server umount lustre-MDT0000 complete [ 5533.716101] LustreError: 316271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033094 with bad export cookie 2727175773155581418 [ 5533.718847] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5533.720742] LustreError: 316271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5533.841493] Lustre: server umount lustre-MDT0001 complete [ 5538.648248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.741595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5538.754353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5538.835489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.135260] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5543.337643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.373769] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5543.384553] Lustre: 318743:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5543.484746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.746038] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5546.902232] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5548.143641] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5552.739143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.814990] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5552.820613] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5552.824032] LustreError: 320493:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5552.826477] LustreError: 320493:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5552.830046] LustreError: 320493:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5552.852995] Lustre: server umount lustre-OST0000 complete [ 5552.854585] LustreError: 320493:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5557.667056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.734755] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5557.796895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.669976] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5561.721321] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5564.280561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.316277] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5564.343344] Lustre: lustre-OST0001: new disk, initializing [ 5564.345686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5564.349093] Lustre: 321960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5566.336777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5566.612760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5566.618316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5566.646037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5566.671586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5568.497055] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5572.066627] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5572.067287] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.069200] LustreError: Skipped 1 previous similar message [ 5572.073434] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5572.076963] Lustre: Skipped 2 previous similar messages [ 5575.548848] Lustre: server umount lustre-OST0001 complete [ 5577.187890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5577.188070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.188861] LustreError: 320961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.188869] LustreError: 320961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5577.190133] LustreError: Skipped 1 previous similar message [ 5577.195671] Lustre: Skipped 1 previous similar message [ 5577.201224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.209383] Lustre: Skipped 1 previous similar message [ 5582.307223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.307377] LustreError: 320962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.309434] Lustre: Skipped 1 previous similar message [ 5582.315249] LustreError: 320962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5582.930871] Lustre: server umount lustre-OST0000 complete [ 5584.352934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5584.356486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.425130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.426874] Lustre: Skipped 2 previous similar messages [ 5590.366792] Lustre: server umount lustre-MDT0000 complete [ 5591.590128] LustreError: 318735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033152 with bad export cookie 2727175773155581964 [ 5591.593400] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.593814] LustreError: 318735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5591.705574] Lustre: server umount lustre-MDT0001 complete [ 5595.328248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5596.271146] Key type lgssc unregistered [ 5596.395305] LNet: 323838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5596.397542] LNetError: 323838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5596.404281] LNet: Removed LNI 192.168.204.115@tcp [ 5596.689397] Key type .llcrypt unregistered [ 5596.690831] Key type ._llcrypt unregistered [ 5626.874120] Key type ._llcrypt registered [ 5626.875212] Key type .llcrypt registered [ 5626.910705] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5633.375563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5633.855358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5633.864325] alg: No test for adler32 (adler32-zlib) [ 5634.725199] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5634.808348] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5636.400251] Key type lgssc registered [ 5636.780863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5647.980463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5647.985620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.074835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5649.086985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5649.125604] Lustre: lustre-MDT0000: new disk, initializing [ 5649.170908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.178102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5650.519407] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5654.983728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.032404] Lustre: 328706:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5655.060627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5655.063596] Lustre: Skipped 1 previous similar message [ 5655.108218] Lustre: lustre-MDT0001: new disk, initializing [ 5655.143896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5655.155855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5655.159024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5656.431745] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5659.791580] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5660.920579] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5663.034264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.136598] Lustre: lustre-OST0000: new disk, initializing [ 5663.139450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5663.143737] Lustre: 330542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5663.178618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.992644] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5668.455858] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5668.854728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5668.857680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5668.891611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5669.802708] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5669.876170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5671.137474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5671.209238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5673.953513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.955594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.956536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.959082] Lustre: Skipped 1 previous similar message [ 5677.905837] Lustre: server umount lustre-OST0000 complete [ 5680.609688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5680.614952] LustreError: Skipped 1 previous similar message [ 5680.618109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.627965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5680.632337] Lustre: Skipped 1 previous similar message [ 5684.193374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.193979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.350546] Lustre: server umount lustre-MDT0000 complete [ 5686.590225] LustreError: 329532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033247 with bad export cookie 11956133370086918894 [ 5686.593897] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.594484] LustreError: 329532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5686.715285] Lustre: server umount lustre-MDT0001 complete [ 5689.450685] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:27:29 (1781033249) [ 5691.836886] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:27:31 (1781033251) [ 5694.571996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.767881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.040065] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5699.141786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.261264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5700.555669] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5702.749917] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5704.057812] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5706.256525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.207783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5710.252887] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5723.839285] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5724.408826] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5724.946858] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5725.528734] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5726.177238] Lustre: Failing over lustre-MDT0000 [ 5726.377396] Lustre: server umount lustre-MDT0000 complete [ 5730.274816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.275418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5730.276243] LustreError: 333195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.276253] LustreError: 333195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5730.279257] Lustre: Skipped 2 previous similar messages [ 5735.393149] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.397630] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5739.229262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.267212] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5739.353350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.355356] Lustre: Skipped 1 previous similar message [ 5739.384698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5740.576103] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5740.986788] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5744.614112] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5744.622334] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5744.640331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5745.756151] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5746.278758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5748.088290] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5748.715132] Lustre: Failing over lustre-MDT0000 [ 5748.873298] Lustre: server umount lustre-MDT0000 complete [ 5749.730743] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.731808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5749.732105] LustreError: 333955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.732120] LustreError: 333955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5749.737028] Lustre: Skipped 2 previous similar messages [ 5754.849699] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.855927] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5759.970612] LustreError: 333955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.975512] LustreError: 333955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5761.700329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.742098] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5761.845587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.877558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5763.161546] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5766.587353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5767.146705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5767.148445] Lustre: Skipped 2 previous similar messages [ 5767.157343] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5767.177034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5768.261129] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5768.766052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5772.258311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.260164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.261993] Lustre: Skipped 2 previous similar messages [ 5772.264081] Lustre: Skipped 2 previous similar messages [ 5772.265515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5777.176866] Lustre: server umount lustre-MDT0000 complete [ 5777.378384] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.386819] LustreError: 333196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5778.382164] LustreError: 333221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033339 with bad export cookie 11956133370086921575 [ 5778.386445] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.387024] LustreError: 333221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5778.507708] Lustre: server umount lustre-MDT0001 complete [ 5789.972738] Lustre: server umount lustre-OST0000 complete [ 5794.932507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.076373] LustreError: 339222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.097383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.354072] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5799.473731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.903217] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5803.045187] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5804.195839] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5806.379554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.399302] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5810.482670] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5819.815217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5824.995650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.996195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5825.000666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5825.002536] LustreError: Skipped 1 previous similar message [ 5825.004078] Lustre: Skipped 1 previous similar message [ 5828.174392] Lustre: server umount lustre-OST0000 complete [ 5835.233310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5835.235783] Lustre: Skipped 3 previous similar messages [ 5835.542780] Lustre: server umount lustre-MDT0000 complete [ 5836.785287] LustreError: 339203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033397 with bad export cookie 11956133370086922296 [ 5836.788320] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.789426] LustreError: 339203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5836.905098] Lustre: server umount lustre-MDT0001 complete [ 5840.503847] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5841.599325] Key type lgssc unregistered [ 5841.750335] LNet: 342561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5841.752856] LNetError: 342561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5841.763415] LNet: Removed LNI 192.168.204.115@tcp [ 5842.071421] Key type .llcrypt unregistered [ 5842.072974] Key type ._llcrypt unregistered [ 5851.951028] Key type ._llcrypt registered [ 5851.952683] Key type .llcrypt registered [ 5851.989919] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5852.297330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5852.308455] alg: No test for adler32 (adler32-zlib) [ 5853.155752] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 5853.233681] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5854.808141] Key type lgssc registered [ 5855.167215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5857.652955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5857.658801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.816679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.062595] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5863.124650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.253578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5864.488955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5866.588316] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5867.817539] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5869.988615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.096184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5871.852335] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5873.905192] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5883.428416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5888.484872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.486308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5888.488335] Lustre: Skipped 2 previous similar messages [ 5888.491491] Lustre: Skipped 1 previous similar message [ 5893.603019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.608285] Lustre: Skipped 2 previous similar messages [ 5894.304420] Lustre: server umount lustre-MDT0000 complete [ 5895.533607] LustreError: 345747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033456 with bad export cookie 9750422575618013725 [ 5895.537388] LustreError: 345747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5895.538571] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5895.663511] Lustre: server umount lustre-MDT0001 complete [ 5907.230389] Lustre: server umount lustre-OST0000 complete [ 5911.662276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.837650] LustreError: 347564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.842932] LustreError: 347564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5911.861668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.212955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5916.484556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.611277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.864625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5919.907369] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5921.053896] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5923.276055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.424532] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5927.617103] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5936.743336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5947.363137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.363348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5947.364359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5947.367428] Lustre: Skipped 1 previous similar message [ 5951.064844] Lustre: server umount lustre-MDT0000 complete [ 5952.237265] LustreError: 349276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033513 with bad export cookie 9750422575618014824 [ 5952.239201] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.240819] LustreError: 349276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5952.353761] Lustre: server umount lustre-MDT0001 complete [ 5963.806727] Lustre: server umount lustre-OST0000 complete [ 5968.826779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.993107] LustreError: 351299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.016036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.018028] Lustre: Skipped 1 previous similar message [ 5970.261865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5973.436625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.936292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5976.963985] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5978.157327] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5980.279224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.189539] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5984.333390] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5994.727568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5999.073625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5999.077186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.083216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5999.085845] Lustre: Skipped 2 previous similar messages [ 6000.099398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6002.318119] Lustre: server umount lustre-OST0000 complete [ 6004.192906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6004.195468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.199510] Lustre: Skipped 1 previous similar message [ 6004.201340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.204094] Lustre: Skipped 2 previous similar messages [ 6009.683887] Lustre: server umount lustre-MDT0000 complete [ 6010.337386] LustreError: 351300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.343047] LustreError: 351300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6010.952445] LustreError: 351281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033571 with bad export cookie 9750422575618015832 [ 6010.956075] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.956238] LustreError: 351281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6011.083955] Lustre: server umount lustre-MDT0001 complete [ 6014.726793] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6015.681075] Key type lgssc unregistered [ 6015.807323] LNet: 354638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6015.809583] LNetError: 354638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6015.820267] LNet: Removed LNI 192.168.204.115@tcp [ 6016.125523] Key type .llcrypt unregistered [ 6016.127247] Key type ._llcrypt unregistered [ 6025.622041] Key type ._llcrypt registered [ 6025.623190] Key type .llcrypt registered [ 6025.659666] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6025.987389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6026.061156] alg: No test for adler32 (adler32-zlib) [ 6026.923245] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6027.011996] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6028.600114] Key type lgssc registered [ 6028.938936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6031.372163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6031.377848] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.380358] LustreError: 356037:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6031.388114] LustreError: 356037:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6031.390387] LustreError: 356037:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6031.392573] LustreError: 356037:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6031.395112] LustreError: 356037:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6033.840635] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.844628] LustreError: 356433:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6033.853927] LustreError: 356433:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6033.857376] LustreError: 356433:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6033.861026] LustreError: 356433:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6033.864171] LustreError: 356433:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6040.117819] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6041.132909] Key type lgssc unregistered [ 6041.249389] LNet: 357418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6041.253283] LNetError: 357418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6041.263321] LNet: Removed LNI 192.168.204.115@tcp [ 6041.537635] Key type .llcrypt unregistered [ 6041.538919] Key type ._llcrypt unregistered [ 6047.452433] Key type ._llcrypt registered [ 6047.453469] Key type .llcrypt registered [ 6047.500420] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6053.423788] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6053.842959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6053.849186] alg: No test for adler32 (adler32-zlib) [ 6054.707896] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6054.792611] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6056.384282] Key type lgssc registered [ 6056.835072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6067.926861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6067.932860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.025324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6069.038470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6069.074352] Lustre: lustre-MDT0000: new disk, initializing [ 6069.097483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.104262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6070.415576] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6074.657270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.700914] Lustre: 360512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.727341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6074.729101] Lustre: Skipped 1 previous similar message [ 6074.770406] Lustre: lustre-MDT0001: new disk, initializing [ 6074.808692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.822843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6074.826140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6076.061529] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6079.366390] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6080.545411] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6082.733843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.836865] Lustre: lustre-OST0000: new disk, initializing [ 6082.838450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6082.840805] Lustre: 362343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6082.870604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.124806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6083.127871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6083.182852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6084.702655] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6088.043696] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6089.289676] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6089.351406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6090.585464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6090.654565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6093.282053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.284398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6093.288194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.289888] Lustre: Skipped 1 previous similar message [ 6093.329763] Lustre: server umount lustre-OST0000 complete [ 6095.328941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6095.331620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.335981] Lustre: Skipped 1 previous similar message [ 6095.338250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.401235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.401794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.697997] Lustre: server umount lustre-MDT0000 complete [ 6101.857334] LustreError: 361338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033662 with bad export cookie 843586098004966096 [ 6101.860465] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6101.860714] LustreError: 361338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6101.978382] Lustre: server umount lustre-MDT0001 complete [ 6104.707442] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6105.265971] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:34:25 (1781033665) [ 6107.367716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.555274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.783600] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6111.799319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.917556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6113.075423] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6114.983443] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6116.054761] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6118.092084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.276621] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6123.001464] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6126.247253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6126.312594] Lustre: lustre-OST0001: new disk, initializing [ 6126.315187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6126.318696] Lustre: 367339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6126.356400] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6126.359084] Lustre: Skipped 1 previous similar message [ 6128.803852] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6128.975481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6128.979940] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6129.015346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6132.632933] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6138.818567] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6138.820493] Lustre: Skipped 1 previous similar message [ 6139.364248] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.364876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6139.369077] Lustre: Skipped 1 previous similar message [ 6139.371563] LustreError: Skipped 1 previous similar message [ 6140.607325] Lustre: server umount lustre-OST0001 complete [ 6144.483937] LustreError: 366389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.489781] LustreError: 366389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6149.604138] LustreError: 366391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.608542] LustreError: 366391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6154.723617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.726071] LustreError: 366391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.730959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.730970] Lustre: Skipped 2 previous similar messages [ 6154.732620] Lustre: Skipped 1 previous similar message [ 6154.740980] LustreError: 366391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6159.848251] LustreError: 367701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.862476] LustreError: 367701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6164.966682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.968116] LustreError: 366390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.969659] Lustre: Skipped 4 previous similar messages [ 6164.977269] LustreError: 366390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6167.008088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6167.073459] Lustre: server umount lustre-OST0000 complete [ 6170.083589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.872902] Lustre: server umount lustre-MDT0000 complete [ 6175.202115] LustreError: 365459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.206497] LustreError: 365459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6176.017240] LustreError: 364683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033736 with bad export cookie 843586098004966908 [ 6176.020977] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6176.022284] LustreError: 364683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6182.201710] Lustre: server umount lustre-MDT0001 complete [ 6185.561750] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6186.524272] Key type lgssc unregistered [ 6186.638392] LNet: 369244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6186.641577] LNetError: 369244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6186.650278] LNet: Removed LNI 192.168.204.115@tcp [ 6186.920847] Key type .llcrypt unregistered [ 6186.921927] Key type ._llcrypt unregistered [ 6205.424274] Key type ._llcrypt registered [ 6205.425320] Key type .llcrypt registered [ 6205.468692] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6205.963433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6205.979732] alg: No test for adler32 (adler32-zlib) [ 6206.836538] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6206.924449] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6208.512261] Key type lgssc registered [ 6209.021800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6211.527542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6211.533529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.644735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6212.654648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6212.717164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.988139] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6216.948155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.981682] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6216.994578] Lustre: 372000:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6217.084441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6218.351802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6220.462751] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6221.709698] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6226.402385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.403211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.405646] Lustre: Skipped 1 previous similar message [ 6228.514470] Lustre: server umount lustre-MDT0000 complete [ 6229.721186] LustreError: 372771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033790 with bad export cookie 4009998313194879678 [ 6229.724338] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.724376] LustreError: 372771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6229.828549] Lustre: server umount lustre-MDT0001 complete [ 6233.024851] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:36:32 (1781033792) [ 6235.178765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.347732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.482124] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6239.340983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.453493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.640516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6242.585773] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6243.697854] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6245.773987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.861272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6245.914478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.562579] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6249.591707] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6258.474122] Lustre: Permanently deactivating lustre-OST0000 [ 6258.479651] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6262.768301] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6264.104877] LustreError: 377167:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6264.107767] LustreError: 377167:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6264.109807] LustreError: 377167:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6264.684774] LustreError: 377216:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6264.688203] LustreError: 377216:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6271.440884] Lustre: server umount lustre-OST0000 complete [ 6272.329581] LustreError: 377466:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6272.333567] LustreError: 377466:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6273.505511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.506214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.509021] Lustre: Skipped 1 previous similar message [ 6273.510557] Lustre: Skipped 1 previous similar message [ 6278.625044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.626711] Lustre: Skipped 2 previous similar messages [ 6283.745101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6283.747051] Lustre: Skipped 1 previous similar message [ 6287.328166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6287.394214] Lustre: server umount lustre-MDT0000 complete [ 6288.764180] LustreError: 374360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033849 with bad export cookie 4009998313194880245 [ 6288.766664] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.767714] LustreError: 374360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6299.155612] Lustre: server umount lustre-MDT0001 complete [ 6302.008059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.386747] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6304.641896] LustreError: 378654:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6306.759721] Lustre: 378994:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.115@tcp [ 6307.425784] Lustre: server umount MGS complete [ 6311.370654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.546311] LustreError: 379716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.551162] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6311.554625] LustreError: 379716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6311.556608] Lustre: Skipped 1 previous similar message [ 6311.577878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.914807] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6316.234381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.342535] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6317.744555] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6319.886049] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6321.100255] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6321.732196] Lustre: Permanently reactivating lustre-OST0000 [ 6321.738577] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6321.741675] Lustre: Skipped 2 previous similar messages [ 6344.160131] LustreError: 381268:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6347.605855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.723630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.725944] Lustre: Skipped 1 previous similar message [ 6349.575840] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6351.695460] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6364.132045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.132175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.136417] LustreError: Skipped 1 previous similar message [ 6364.143549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.147382] Lustre: Skipped 1 previous similar message [ 6365.386682] Lustre: server umount lustre-OST0000 complete [ 6367.713167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6367.716183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.720928] Lustre: Skipped 1 previous similar message [ 6368.670726] Lustre: server umount lustre-MDT0000 complete [ 6369.249651] LustreError: 379735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.259459] LustreError: 379735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6370.101565] LustreError: 379697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781033930 with bad export cookie 4009998313194881267 [ 6370.105923] LustreError: 379697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6370.106178] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6370.235279] Lustre: server umount lustre-MDT0001 complete [ 6373.993112] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6375.175775] Key type lgssc unregistered [ 6375.314356] LNet: 384075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6375.318465] LNetError: 384075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6375.329310] LNet: Removed LNI 192.168.204.115@tcp [ 6375.775371] Key type .llcrypt unregistered [ 6375.778912] Key type ._llcrypt unregistered [ 6382.364133] Key type ._llcrypt registered [ 6382.365147] Key type .llcrypt registered [ 6382.410767] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6388.643824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6389.070472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6389.082308] alg: No test for adler32 (adler32-zlib) [ 6389.935496] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6390.039909] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6391.632142] Key type lgssc registered [ 6392.043044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.438615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.450344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.549386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.559921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6405.598421] Lustre: lustre-MDT0000: new disk, initializing [ 6405.621472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.629709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6406.928391] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6411.829359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.869446] Lustre: 387171:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6411.890308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6411.892031] Lustre: Skipped 1 previous similar message [ 6411.925084] Lustre: lustre-MDT0001: new disk, initializing [ 6411.954457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6411.966114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6411.970152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6413.196937] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6416.669354] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6417.843082] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6420.167519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.281964] Lustre: lustre-OST0000: new disk, initializing [ 6420.284785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6420.288483] Lustre: 389010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6420.316934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.663267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6422.667240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6422.708714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6422.716014] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6426.191440] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6427.783496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6427.864056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6429.496683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6429.566156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6433.252605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.252839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6433.253943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6433.256487] Lustre: Skipped 1 previous similar message [ 6433.258633] LustreError: Skipped 1 previous similar message [ 6436.384459] Lustre: server umount lustre-OST0000 complete [ 6437.863797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6437.869686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.877932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.880640] Lustre: Skipped 1 previous similar message [ 6443.489748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6443.493910] Lustre: Skipped 3 previous similar messages [ 6448.609458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.612407] Lustre: Skipped 1 previous similar message [ 6452.192097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6452.253226] Lustre: server umount lustre-MDT0000 complete [ 6453.444029] LustreError: 387163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034014 with bad export cookie 12213588441521872891 [ 6453.447105] LustreError: 387163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6453.447289] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6453.562349] Lustre: server umount lustre-MDT0001 complete [ 6456.389778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6456.891795] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:40:16 (1781034016) [ 6459.015706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.202206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.514528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6463.566643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.677758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6464.887622] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6466.925047] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6468.093811] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6470.347632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6472.288334] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6474.315787] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6474.913810] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6485.519635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6489.065346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6489.065603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.072577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.084879] Lustre: Skipped 2 previous similar messages [ 6493.677467] Lustre: server umount lustre-OST0000 complete [ 6499.299145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.299994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6499.308738] Lustre: Skipped 1 previous similar message [ 6499.313058] Lustre: Skipped 1 previous similar message [ 6501.626927] Lustre: server umount lustre-MDT0000 complete [ 6503.204310] LustreError: 391352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034063 with bad export cookie 12213588441521873773 [ 6503.210219] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.211980] LustreError: 391352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6503.383629] Lustre: server umount lustre-MDT0001 complete [ 6507.365123] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6508.606464] Key type lgssc unregistered [ 6508.752564] LNet: 394902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6508.755643] LNetError: 394902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6508.769329] LNet: Removed LNI 192.168.204.115@tcp [ 6509.137826] Key type .llcrypt unregistered [ 6509.139777] Key type ._llcrypt unregistered [ 6518.743559] Key type ._llcrypt registered [ 6518.744571] Key type .llcrypt registered [ 6518.782320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6518.883222] Key type .llcrypt unregistered [ 6518.884150] Key type ._llcrypt unregistered [ 6526.166201] Key type ._llcrypt registered [ 6526.167151] Key type .llcrypt registered [ 6526.202859] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6526.717794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6526.728366] alg: No test for adler32 (adler32-zlib) [ 6527.614646] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6527.709065] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6529.288144] Key type lgssc registered [ 6529.738921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6532.576843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6532.582805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.768890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6535.128235] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6538.412784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.520449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.296484] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6542.882364] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6543.651103] LustreError: 397055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.656500] LustreError: 397055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6543.659981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6545.623789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.759691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6547.036457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6556.132113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.136255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6556.136597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6556.142083] Lustre: Skipped 1 previous similar message [ 6560.778605] Lustre: server umount lustre-OST0000 complete [ 6565.858427] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6565.861558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.865372] Lustre: Skipped 1 previous similar message [ 6565.867154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.087623] Lustre: server umount lustre-MDT0000 complete [ 6569.240120] LustreError: 397036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034130 with bad export cookie 11698541793334275983 [ 6569.243240] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.246287] LustreError: 397036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6569.366664] Lustre: server umount lustre-MDT0001 complete [ 6573.069491] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6574.068740] Key type lgssc unregistered [ 6574.198352] LNet: 400184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6574.200964] LNetError: 400184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6574.209252] LNet: Removed LNI 192.168.204.115@tcp [ 6574.505788] Key type .llcrypt unregistered [ 6574.507777] Key type ._llcrypt unregistered [ 6583.683991] Key type ._llcrypt registered [ 6583.685125] Key type .llcrypt registered [ 6583.718636] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6584.150153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6584.161782] alg: No test for adler32 (adler32-zlib) [ 6585.006865] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6585.085916] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6586.664143] Key type lgssc registered [ 6587.019736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6590.021859] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6590.031127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.536429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6610.656234] LustreError: 401533:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 6637.280254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.082317] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6641.074599] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6643.206133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.395070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.419794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6644.682119] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6647.737718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.866788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6649.119257] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6651.227276] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6652.382319] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6656.998284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.999147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6657.007140] Lustre: Skipped 1 previous similar message [ 6660.761600] Lustre: server umount lustre-OST0000 complete [ 6662.114388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.115549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.120414] Lustre: Skipped 1 previous similar message [ 6662.123656] Lustre: Skipped 2 previous similar messages [ 6667.233448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.235429] Lustre: Skipped 1 previous similar message [ 6668.125780] Lustre: server umount lustre-MDT0000 complete [ 6669.388453] LustreError: 401540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034230 with bad export cookie 13737720894477688481 [ 6669.393720] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.521104] Lustre: server umount lustre-MDT0001 complete [ 6673.087120] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6674.186981] Key type lgssc unregistered [ 6674.324470] LNet: 404891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6674.326971] LNetError: 404891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6674.334392] LNet: Removed LNI 192.168.204.115@tcp [ 6674.638247] Key type .llcrypt unregistered [ 6674.640077] Key type ._llcrypt unregistered [ 6683.309636] Key type ._llcrypt registered [ 6683.310720] Key type .llcrypt registered [ 6683.344873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6683.853862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6683.860425] alg: No test for adler32 (adler32-zlib) [ 6684.718483] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6684.802615] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6686.392137] Key type lgssc registered [ 6686.754161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6689.338430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6689.345437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.519268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.769380] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6694.744269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.868969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.174625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6698.119893] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6699.218494] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6701.342077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.471736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.520892] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6706.384514] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6715.818666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6720.481177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6720.484314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.489405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6720.996875] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6721.006568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.010663] Lustre: Skipped 1 previous similar message [ 6724.214239] Lustre: server umount lustre-MDT0000 complete [ 6725.212258] LustreError: 408672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1781034285 with bad export cookie 9989671675077647466 [ 6725.217791] LustreError: 408672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6726.114681] LustreError: 406268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.124157] LustreError: 406268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6726.137721] LustreError: 407977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034286 with bad export cookie 9989671675077646815 [ 6726.144914] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.145393] LustreError: 407977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6730.209884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6730.214883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.222291] Lustre: Skipped 1 previous similar message [ 6730.225779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6730.229792] Lustre: Skipped 1 previous similar message [ 6731.233799] LustreError: 407037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.244221] LustreError: 407037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6732.271596] Lustre: server umount lustre-OST0000 complete [ 6735.097879] Lustre: server umount lustre-MDT0001 complete [ 6739.226235] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6740.545916] Key type lgssc unregistered [ 6740.670376] LNet: 409761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6740.672564] LNetError: 409761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6740.684302] LNet: Removed LNI 192.168.204.115@tcp [ 6740.993346] Key type .llcrypt unregistered [ 6740.994850] Key type ._llcrypt unregistered [ 6750.097854] Key type ._llcrypt registered [ 6750.099430] Key type .llcrypt registered [ 6750.135076] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6750.454335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6750.488083] alg: No test for adler32 (adler32-zlib) [ 6751.335733] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6751.414697] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6752.992219] Key type lgssc registered [ 6753.454179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6756.410994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6756.422834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.624455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.077151] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6762.604346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.735340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6764.194439] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6767.116914] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6768.328563] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6770.704559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6770.840215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.839143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6775.330002] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6783.144655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6788.578496] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.582240] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6788.583477] Lustre: Skipped 1 previous similar message [ 6791.104834] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6791.107716] Lustre: Skipped 1 previous similar message [ 6793.698092] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6793.701041] Lustre: Skipped 1 previous similar message [ 6796.218592] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6801.340570] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6801.345953] Lustre: Skipped 2 previous similar messages [ 6801.376157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6801.462205] Lustre: server umount lustre-MDT0001 complete [ 6803.937463] LustreError: 411906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6803.938034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6803.938039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.941626] LustreError: 411906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6809.057784] LustreError: 411136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.368490] Lustre: server umount lustre-OST0000 complete [ 6810.688095] Lustre: server umount lustre-MDT0000 complete [ 6815.010849] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6816.131594] Key type lgssc unregistered [ 6816.305650] LNet: 414630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6816.309615] LNetError: 414630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6816.321524] LNet: Removed LNI 192.168.204.115@tcp [ 6816.728536] Key type .llcrypt unregistered [ 6816.730375] Key type ._llcrypt unregistered [ 6825.724660] Key type ._llcrypt registered [ 6825.726189] Key type .llcrypt registered [ 6825.759887] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6832.224332] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6832.548320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.604074] alg: No test for adler32 (adler32-zlib) [ 6833.449523] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6833.529072] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6835.104130] Key type lgssc registered [ 6835.444816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6847.131563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6852.366128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6852.372816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.474155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6853.484960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6853.530130] Lustre: lustre-MDT0000: new disk, initializing [ 6853.555587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.563355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6854.819565] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6861.672430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6861.786188] Lustre: lustre-OST0000: new disk, initializing [ 6861.789346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6861.793066] Lustre: Skipped 1 previous similar message [ 6861.796567] Lustre: 419613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6861.822356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.600356] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6867.167416] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6870.524243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6870.526671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6870.533393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6878.515696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.563047] Lustre: 418565:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.593635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6878.640905] Lustre: lustre-MDT0001: new disk, initializing [ 6878.688129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.703195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6878.707806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6880.152779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6884.342817] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6884.435117] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6885.872376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6885.979712] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6888.900580] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6889.956922] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6889.961129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.963146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6889.971322] Lustre: Skipped 1 previous similar message [ 6894.011062] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6894.014096] Lustre: Skipped 1 previous similar message [ 6899.130649] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6899.133414] Lustre: Skipped 2 previous similar messages [ 6902.240127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6902.298060] Lustre: server umount lustre-MDT0001 complete [ 6910.434224] LustreError: 418572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6910.439936] LustreError: 418572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6915.553799] LustreError: 418575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6915.554082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.561599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.563175] Lustre: Skipped 2 previous similar messages [ 6920.654433] Lustre: server umount lustre-OST0000 complete [ 6920.675023] LustreError: 418576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6928.184111] Lustre: server umount lustre-MDT0000 complete [ 6932.949579] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6934.294744] Key type lgssc unregistered [ 6934.449582] LNet: 423273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6934.454239] LNetError: 423273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6934.462610] LNet: Removed LNI 192.168.204.115@tcp [ 6934.854301] Key type .llcrypt unregistered [ 6934.856041] Key type ._llcrypt unregistered [ 6950.146794] Key type ._llcrypt registered [ 6950.147890] Key type .llcrypt registered [ 6950.187464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6950.282273] Key type .llcrypt unregistered [ 6950.283162] Key type ._llcrypt unregistered [ 6957.024516] Key type ._llcrypt registered [ 6957.025548] Key type .llcrypt registered [ 6957.056904] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6957.370794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6957.385087] alg: No test for adler32 (adler32-zlib) [ 6958.249348] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 6958.350568] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6959.936162] Key type lgssc registered [ 6960.391707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6970.528473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6970.533654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.643528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6971.655820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6971.700282] Lustre: lustre-MDT0000: new disk, initializing [ 6971.723397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.731062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6973.051587] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6979.046675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.130833] Lustre: 426589:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6979.174137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6979.178073] Lustre: Skipped 1 previous similar message [ 6979.236383] Lustre: lustre-MDT0001: new disk, initializing [ 6979.280812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6979.300839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6979.308585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6980.934545] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6985.243091] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6986.803066] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6989.606954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6989.733835] Lustre: lustre-OST0000: new disk, initializing [ 6989.735922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6989.739905] Lustre: 428429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.777547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6991.621783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6992.317757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6992.320891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6992.354572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6995.467721] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7000.033592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7000.038348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.046584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7002.594959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7002.596943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.598084] Lustre: Skipped 1 previous similar message [ 7002.605177] Lustre: Skipped 1 previous similar message [ 7004.323918] Lustre: server umount lustre-MDT0000 complete [ 7005.755705] LustreError: 427415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034566 with bad export cookie 1121307907360577147 [ 7005.758859] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7005.761377] LustreError: 427415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7005.908268] Lustre: server umount lustre-MDT0001 complete [ 7017.636720] Lustre: server umount lustre-OST0000 complete [ 7021.953147] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:49:41 (1781034581) [ 7025.748991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.976350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.836952] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7031.267298] LustreError: 430390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.279176] LustreError: 430390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7032.550025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.690559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.628791] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7037.866587] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7039.683570] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7056.165301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.274197] Lustre: Found index 0 for lustre-OST0000, updating log [ 7056.362874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.062750] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7062.072829] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7071.732509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7076.840626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.840919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.845901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.860290] Lustre: Skipped 1 previous similar message [ 7080.356107] Lustre: server umount lustre-OST0000 complete [ 7094.241687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7094.242433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7094.242619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7094.242625] Lustre: Skipped 3 previous similar messages [ 7094.248506] Lustre: Skipped 1 previous similar message [ 7099.364409] LustreError: 431160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.372555] LustreError: 431160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7099.388615] Lustre: server umount lustre-MDT0000 complete [ 7101.030964] LustreError: 430373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781034661 with bad export cookie 1121307907360578540 [ 7101.037698] LustreError: 430373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7101.038572] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.192623] Lustre: server umount lustre-MDT0001 complete [ 7104.820111] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:51:04 (1781034664) [ 7119.136442] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [ 7688.130197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.381185] LustreError: 436619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.409135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7690.342390] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7693.794501] LustreError: 436620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.818812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.005800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7696.860993] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7700.059503] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7701.671643] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7717.413576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.559311] Lustre: Found index 0 for lustre-OST0000, updating log [ 7717.665842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.981393] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7722.965570] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7728.098860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7728.100365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.109245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7728.112315] Lustre: Skipped 1 previous similar message [ 7729.960649] Lustre: server umount lustre-OST0000 complete [ 7732.834070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.835256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.838466] Lustre: Skipped 1 previous similar message [ 7733.857420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7733.859282] Lustre: Skipped 2 previous similar messages [ 7735.907218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.910983] Lustre: Skipped 3 previous similar messages [ 7737.957586] LustreError: 437379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.966654] LustreError: 437379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7737.988217] Lustre: server umount lustre-MDT0000 complete [ 7739.643772] LustreError: 436600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035300 with bad export cookie 1121307907360579625 [ 7739.648457] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.650911] LustreError: 436600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7739.802882] Lustre: server umount lustre-MDT0001 complete [ 7747.872472] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [ 7759.412393] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7762.953912] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7776.566527] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:02:16 (1781035336) [ 7783.235781] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7784.469557] Key type lgssc unregistered [ 7784.586460] LNet: 443046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.589061] LNetError: 443046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7849.126683] LNet: Removed LNI 192.168.204.115@tcp [ 7849.635185] Key type .llcrypt unregistered [ 7849.637534] Key type ._llcrypt unregistered [ 7858.714833] Key type ._llcrypt registered [ 7858.716106] Key type .llcrypt registered [ 7858.789446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7859.270202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7859.330122] alg: No test for adler32 (adler32-zlib) [ 7860.217661] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 7860.347083] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7861.936288] Key type lgssc registered [ 7862.607481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7864.933433] LNet: Added LNI 192.168.204.115@tcp100 [8/256/0/180] [ 7865.392822] LNet: Added LNI 192.168.204.115@tcp101 [8/256/0/180] [ 7865.843558] LNet: Added LNI 192.168.204.115@tcp102 [8/256/0/180] [ 7866.307831] LNet: Added LNI 192.168.204.115@tcp103 [8/256/0/180] [ 7866.758289] LNet: Added LNI 192.168.204.115@tcp104 [8/256/0/180] [ 7867.191906] LNet: Added LNI 192.168.204.115@tcp105 [8/256/0/180] [ 7867.644798] LNet: Added LNI 192.168.204.115@tcp106 [8/256/0/180] [ 7868.103216] LNet: Added LNI 192.168.204.115@tcp107 [8/256/0/180] [ 7868.550482] LNet: Added LNI 192.168.204.115@tcp108 [8/256/0/180] [ 7868.996911] LNet: Added LNI 192.168.204.115@tcp109 [8/256/0/180] [ 7869.440815] LNet: Added LNI 192.168.204.115@tcp110 [8/256/0/180] [ 7869.909121] LNet: Added LNI 192.168.204.115@tcp111 [8/256/0/180] [ 7870.331833] LNet: Added LNI 192.168.204.115@tcp112 [8/256/0/180] [ 7870.750556] LNet: Added LNI 192.168.204.115@tcp113 [8/256/0/180] [ 7871.172613] LNet: Added LNI 192.168.204.115@tcp114 [8/256/0/180] [ 7871.612732] LNet: Added LNI 192.168.204.115@tcp115 [8/256/0/180] [ 7872.018883] LNet: Added LNI 192.168.204.115@tcp116 [8/256/0/180] [ 7872.442601] LNet: Added LNI 192.168.204.115@tcp117 [8/256/0/180] [ 7872.859586] LNet: Added LNI 192.168.204.115@tcp118 [8/256/0/180] [ 7873.269852] LNet: Added LNI 192.168.204.115@tcp119 [8/256/0/180] [ 7873.693073] LNet: Added LNI 192.168.204.115@tcp120 [8/256/0/180] [ 7874.117464] LNet: Added LNI 192.168.204.115@tcp121 [8/256/0/180] [ 7874.499085] LNet: Added LNI 192.168.204.115@tcp122 [8/256/0/180] [ 7874.908782] LNet: Added LNI 192.168.204.115@tcp123 [8/256/0/180] [ 7875.281218] LNet: Added LNI 192.168.204.115@tcp124 [8/256/0/180] [ 7875.639805] LNet: Added LNI 192.168.204.115@tcp125 [8/256/0/180] [ 7875.958201] LNet: Added LNI 192.168.204.115@tcp126 [8/256/0/180] [ 7876.353034] LNet: Added LNI 192.168.204.115@tcp127 [8/256/0/180] [ 7876.711295] LNet: Added LNI 192.168.204.115@tcp128 [8/256/0/180] [ 7877.084725] LNet: Added LNI 192.168.204.115@tcp129 [8/256/0/180] [ 7877.517112] LNet: Added LNI 192.168.204.115@tcp130 [8/256/0/180] [ 7877.958928] LNet: Added LNI 192.168.204.115@tcp131 [8/256/0/180] [ 7878.370482] LNet: Added LNI 192.168.204.115@tcp132 [8/256/0/180] [ 7878.837955] LNet: Added LNI 192.168.204.115@tcp133 [8/256/0/180] [ 7879.300909] LNet: Added LNI 192.168.204.115@tcp134 [8/256/0/180] [ 7879.771623] LNet: Added LNI 192.168.204.115@tcp135 [8/256/0/180] [ 7882.913987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7882.922489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.002178] Lustre: 446026:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 7884.100853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7884.116826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7884.182765] Lustre: lustre-MDT0000: new disk, initializing [ 7884.243810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.258451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7886.258236] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7892.191128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.274633] Lustre: 446043:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7892.318912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7892.322317] Lustre: Skipped 1 previous similar message [ 7892.366535] Lustre: lustre-MDT0001: new disk, initializing [ 7892.378300] Lustre: 446876:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 7892.380728] Lustre: 446876:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7892.429686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7892.449467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7892.454859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7893.939897] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7898.583413] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7899.145649] Lustre: 447447:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 7899.152193] Lustre: 447447:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7900.351400] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7903.812773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.015997] Lustre: lustre-OST0000: new disk, initializing [ 7904.020061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7904.025291] Lustre: 447887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7904.076258] Lustre: 447892:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [ 7904.078854] Lustre: 447892:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7904.095517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7906.493348] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7907.673690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7907.679462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7907.737322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7910.626795] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7918.054970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7918.055454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.060358] LustreError: Skipped 1 previous similar message [ 7918.061545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.071231] Lustre: Skipped 1 previous similar message [ 7923.175055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.179160] Lustre: Skipped 2 previous similar messages [ 7928.288224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7928.294079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7928.303603] Lustre: Skipped 1 previous similar message [ 7928.370898] Lustre: server umount lustre-OST0000 complete [ 7933.413649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7933.414089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.419731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.427175] Lustre: Skipped 1 previous similar message [ 7938.530838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7938.534639] Lustre: Skipped 2 previous similar messages [ 7944.160164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7944.262258] Lustre: server umount lustre-MDT0000 complete [ 7945.843782] LustreError: 446036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035506 with bad export cookie 5076481328187329188 [ 7945.852303] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.034785] Lustre: server umount lustre-MDT0001 complete [ 7950.904346] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7952.421911] Key type lgssc unregistered [ 7952.621649] LNet: 449719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7952.628719] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7952.634128] LNet: Removed LNI 192.168.204.115@tcp [ 7953.696282] LNet: Removed LNI 192.168.204.115@tcp100 [ 7953.701114] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7953.708810] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7953.717539] LNet: Removed LNI 192.168.204.115@tcp101 [ 7954.784246] LNet: Removed LNI 192.168.204.115@tcp102 [ 7954.787269] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7954.791877] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7954.796878] LNet: Removed LNI 192.168.204.115@tcp103 [ 7955.808201] LNet: Removed LNI 192.168.204.115@tcp104 [ 7956.832248] LNet: Removed LNI 192.168.204.115@tcp105 [ 7956.835140] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.839357] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7956.843621] LNet: Removed LNI 192.168.204.115@tcp106 [ 7957.856211] LNet: Removed LNI 192.168.204.115@tcp107 [ 7958.880209] LNet: Removed LNI 192.168.204.115@tcp108 [ 7959.904237] LNet: Removed LNI 192.168.204.115@tcp109 [ 7960.928207] LNet: Removed LNI 192.168.204.115@tcp110 [ 7960.930928] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7960.935644] LNetError: 449719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7960.940249] LNet: Removed LNI 192.168.204.115@tcp111 [ 7961.952180] LNet: Removed LNI 192.168.204.115@tcp112 [ 7962.976170] LNet: Removed LNI 192.168.204.115@tcp113 [ 7964.000139] LNet: Removed LNI 192.168.204.115@tcp114 [ 7965.024212] LNet: Removed LNI 192.168.204.115@tcp115 [ 7965.026927] LNet: Removed LNI 192.168.204.115@tcp116 [ 7965.029657] LNet: Removed LNI 192.168.204.115@tcp117 [ 7965.032366] LNet: Removed LNI 192.168.204.115@tcp118 [ 7965.034919] LNet: Removed LNI 192.168.204.115@tcp119 [ 7965.037577] LNet: Removed LNI 192.168.204.115@tcp120 [ 7965.040069] LNet: Removed LNI 192.168.204.115@tcp121 [ 7965.042488] LNet: Removed LNI 192.168.204.115@tcp122 [ 7965.044862] LNet: Removed LNI 192.168.204.115@tcp123 [ 7965.046981] LNet: Removed LNI 192.168.204.115@tcp124 [ 7965.049347] LNet: Removed LNI 192.168.204.115@tcp125 [ 7965.051802] LNet: Removed LNI 192.168.204.115@tcp126 [ 7965.054197] LNet: Removed LNI 192.168.204.115@tcp127 [ 7965.056323] LNet: Removed LNI 192.168.204.115@tcp128 [ 7965.058646] LNet: Removed LNI 192.168.204.115@tcp129 [ 7965.060873] LNet: Removed LNI 192.168.204.115@tcp130 [ 7965.063225] LNet: Removed LNI 192.168.204.115@tcp131 [ 7965.065426] LNet: Removed LNI 192.168.204.115@tcp132 [ 7965.067819] LNet: Removed LNI 192.168.204.115@tcp133 [ 7965.070204] LNet: Removed LNI 192.168.204.115@tcp134 [ 7965.078799] LNet: Removed LNI 192.168.204.115@tcp135 [ 7965.493952] Key type .llcrypt unregistered [ 7965.495785] Key type ._llcrypt unregistered [ 7975.418717] Key type ._llcrypt registered [ 7975.420047] Key type .llcrypt registered [ 7975.480373] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7983.183777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7983.756131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7983.763425] alg: No test for adler32 (adler32-zlib) [ 7984.645409] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 7984.775614] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7986.384236] Key type lgssc registered [ 7986.904730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8003.852924] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:06:03 (1781035563) [ 8011.056277] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8012.175765] Key type lgssc unregistered [ 8012.305326] LNet: 453726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8012.307572] LNetError: 453726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8012.313346] LNet: Removed LNI 192.168.204.115@tcp [ 8012.595456] Key type .llcrypt unregistered [ 8012.596535] Key type ._llcrypt unregistered [ 8020.218947] Key type ._llcrypt registered [ 8020.221165] Key type .llcrypt registered [ 8020.273000] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8020.674399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8020.732101] alg: No test for adler32 (adler32-zlib) [ 8021.645931] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 8021.769908] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8023.368241] Key type lgssc registered [ 8023.902193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8028.103962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8028.115492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8029.237171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8029.249938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8029.292398] Lustre: lustre-MDT0000: new disk, initializing [ 8029.321249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.329707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8030.801496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8035.450922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.519152] Lustre: 454899:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8035.562630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8035.565982] Lustre: Skipped 1 previous similar message [ 8035.611574] Lustre: lustre-MDT0001: new disk, initializing [ 8035.651377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8035.667366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8035.670576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8037.300329] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8041.482098] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8042.922300] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8046.116895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8046.279589] Lustre: lustre-OST0000: new disk, initializing [ 8046.282985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8046.288019] Lustre: 456739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8046.333551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.946968] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8051.196682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8051.199789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8051.238396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8052.861870] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8054.486362] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 8056.418183] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8056.427790] LNet: Added LNI 192.168.204.115@tcp10 [8/256/0/180] [ 8058.761993] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8058.780629] LNet: Added LNI 192.168.204.115@tcp11 [8/256/0/180] [ 8060.641372] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8060.654878] LNet: Added LNI 192.168.204.115@tcp12 [8/256/0/180] [ 8062.386108] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8062.399366] LNet: Added LNI 192.168.204.115@tcp13 [8/256/0/180] [ 8064.175514] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8064.185860] LNet: Added LNI 192.168.204.115@tcp14 [8/256/0/180] [ 8066.052500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8066.069865] LNet: Added LNI 192.168.204.115@tcp15 [8/256/0/180] [ 8067.886504] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8067.902468] LNet: Added LNI 192.168.204.115@tcp16 [8/256/0/180] [ 8069.630125] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8069.641665] LNet: Added LNI 192.168.204.115@tcp17 [8/256/0/180] [ 8071.226233] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8071.238615] LNet: Added LNI 192.168.204.115@tcp18 [8/256/0/180] [ 8072.584359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8072.593053] LNet: Added LNI 192.168.204.115@tcp19 [8/256/0/180] [ 8073.929625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8073.939217] LNet: Added LNI 192.168.204.115@tcp20 [8/256/0/180] [ 8075.589320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8075.600024] LNet: Added LNI 192.168.204.115@tcp21 [8/256/0/180] [ 8077.416240] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8077.430024] LNet: Added LNI 192.168.204.115@tcp22 [8/256/0/180] [ 8079.401854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8079.414326] LNet: Added LNI 192.168.204.115@tcp23 [8/256/0/180] [ 8081.599232] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8081.618098] LNet: Added LNI 192.168.204.115@tcp24 [8/256/0/180] [ 8083.842474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8083.861748] LNet: Added LNI 192.168.204.115@tcp25 [8/256/0/180] [ 8085.674140] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8085.683709] LNet: Added LNI 192.168.204.115@tcp26 [8/256/0/180] [ 8087.603723] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8087.614708] LNet: Added LNI 192.168.204.115@tcp27 [8/256/0/180] [ 8089.576562] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8089.592925] LNet: Added LNI 192.168.204.115@tcp28 [8/256/0/180] [ 8091.810474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8091.821119] LNet: Added LNI 192.168.204.115@tcp29 [8/256/0/180] [ 8093.921272] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8093.932282] LNet: Added LNI 192.168.204.115@tcp30 [8/256/0/180] [ 8095.976787] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8095.986752] LNet: Added LNI 192.168.204.115@tcp31 [8/256/0/180] [ 8098.103624] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8098.115710] LNet: Added LNI 192.168.204.115@tcp32 [8/256/0/180] [ 8100.321918] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8100.339834] LNet: Added LNI 192.168.204.115@tcp33 [8/256/0/180] [ 8103.427363] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8103.447378] LNet: Added LNI 192.168.204.115@tcp34 [8/256/0/180] [ 8106.559261] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8106.579598] LNet: Added LNI 192.168.204.115@tcp35 [8/256/0/180] [ 8109.638466] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8109.658646] LNet: Added LNI 192.168.204.115@tcp36 [8/256/0/180] [ 8112.777291] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8112.797212] LNet: Added LNI 192.168.204.115@tcp37 [8/256/0/180] [ 8115.699714] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8115.717697] LNet: Added LNI 192.168.204.115@tcp38 [8/256/0/180] [ 8118.285165] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8118.296193] LNet: Added LNI 192.168.204.115@tcp39 [8/256/0/180] [ 8121.068235] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8121.088470] LNet: Added LNI 192.168.204.115@tcp40 [8/256/0/180] [ 8124.277374] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8124.291047] LNet: Added LNI 192.168.204.115@tcp41 [8/256/0/180] [ 8126.980445] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8127.000607] LNet: Added LNI 192.168.204.115@tcp42 [8/256/0/180] [ 8130.075952] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8130.086275] LNet: Added LNI 192.168.204.115@tcp43 [8/256/0/180] [ 8132.694539] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8132.705126] LNet: Added LNI 192.168.204.115@tcp44 [8/256/0/180] [ 8135.469412] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8135.482231] LNet: Added LNI 192.168.204.115@tcp45 [8/256/0/180] [ 8138.388163] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8138.406436] LNet: Added LNI 192.168.204.115@tcp46 [8/256/0/180] [ 8141.737004] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8141.751102] LNet: Added LNI 192.168.204.115@tcp47 [8/256/0/180] [ 8145.274432] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8145.291447] LNet: Added LNI 192.168.204.115@tcp48 [8/256/0/180] [ 8148.857147] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8148.875128] LNet: Added LNI 192.168.204.115@tcp49 [8/256/0/180] [ 8152.076327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8152.088064] LNet: Added LNI 192.168.204.115@tcp50 [8/256/0/180] [ 8163.209867] Lustre: server umount lustre-MDT0000 complete [ 8163.816185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8163.818903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.823036] Lustre: Skipped 2 previous similar messages [ 8165.168499] LustreError: 456738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035725 with bad export cookie 3416902656050261219 [ 8165.176145] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8165.176584] LustreError: 456738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8165.339657] Lustre: server umount lustre-MDT0001 complete [ 8175.240609] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8179.575420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.288214] Lustre: 454175:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781035729/real 1781035729] req@00000000fdda7d17 x1867551166519168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781035745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8184.305593] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.336229] Lustre: 454176:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781035731/real 1781035731] req@0000000021bd2640 x1867551166519296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781035747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8189.408236] Lustre: 454174:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781035734/real 1781035734] req@0000000022ddbdf6 x1867551166519424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781035750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8189.472330] LustreError: 454172:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000bd0f073 x1867551166520320/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8189.597464] LustreError: 456730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.616597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.617942] LustreError: 456730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8191.518426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8194.722935] LustreError: 480130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.733436] LustreError: 480130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8195.744385] Lustre: 454175:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781035740/real 1781035740] req@00000000be0a2b79 x1867551166519808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781035756 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8195.761825] Lustre: 454175:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8196.180018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.396683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8198.197625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8201.155908] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8201.704519] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8202.706406] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8203.750358] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8213.990925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.992247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8214.001124] Lustre: Skipped 1 previous similar message [ 8218.736198] Lustre: server umount lustre-OST0000 complete [ 8222.177648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8222.182992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8222.192308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.196268] Lustre: Skipped 1 previous similar message [ 8224.226325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.230378] Lustre: Skipped 1 previous similar message [ 8226.747105] Lustre: server umount lustre-MDT0000 complete [ 8228.558184] LustreError: 455725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035789 with bad export cookie 3416902656050263319 [ 8228.568810] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.728564] Lustre: server umount lustre-MDT0001 complete [ 8233.898559] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8235.561472] Key type lgssc unregistered [ 8235.804588] LNet: 482758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8235.811350] LNetError: 482758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8235.815966] LNet: Removed LNI 192.168.204.115@tcp [ 8235.818591] LNet: Removed LNI 192.168.204.115@tcp10 [ 8235.821578] LNet: Removed LNI 192.168.204.115@tcp11 [ 8235.824141] LNet: Removed LNI 192.168.204.115@tcp12 [ 8235.826654] LNet: Removed LNI 192.168.204.115@tcp13 [ 8235.829196] LNet: Removed LNI 192.168.204.115@tcp14 [ 8235.831668] LNet: Removed LNI 192.168.204.115@tcp15 [ 8235.834201] LNet: Removed LNI 192.168.204.115@tcp16 [ 8235.836688] LNet: Removed LNI 192.168.204.115@tcp17 [ 8235.839280] LNet: Removed LNI 192.168.204.115@tcp18 [ 8235.841765] LNet: Removed LNI 192.168.204.115@tcp19 [ 8235.844287] LNet: Removed LNI 192.168.204.115@tcp20 [ 8235.846753] LNet: Removed LNI 192.168.204.115@tcp21 [ 8235.849288] LNet: Removed LNI 192.168.204.115@tcp22 [ 8235.851770] LNet: Removed LNI 192.168.204.115@tcp23 [ 8235.854374] LNet: Removed LNI 192.168.204.115@tcp24 [ 8235.856832] LNet: Removed LNI 192.168.204.115@tcp25 [ 8235.859386] LNet: Removed LNI 192.168.204.115@tcp26 [ 8235.861868] LNet: Removed LNI 192.168.204.115@tcp27 [ 8235.864351] LNet: Removed LNI 192.168.204.115@tcp28 [ 8235.866657] LNet: Removed LNI 192.168.204.115@tcp29 [ 8235.868987] LNet: Removed LNI 192.168.204.115@tcp30 [ 8235.871408] LNet: Removed LNI 192.168.204.115@tcp31 [ 8235.873763] LNet: Removed LNI 192.168.204.115@tcp32 [ 8235.875922] LNet: Removed LNI 192.168.204.115@tcp33 [ 8235.878126] LNet: Removed LNI 192.168.204.115@tcp34 [ 8235.880093] LNet: Removed LNI 192.168.204.115@tcp35 [ 8235.882108] LNet: Removed LNI 192.168.204.115@tcp36 [ 8235.884114] LNet: Removed LNI 192.168.204.115@tcp37 [ 8235.886082] LNet: Removed LNI 192.168.204.115@tcp38 [ 8235.887976] LNet: Removed LNI 192.168.204.115@tcp39 [ 8235.889974] LNet: Removed LNI 192.168.204.115@tcp40 [ 8235.891802] LNet: Removed LNI 192.168.204.115@tcp41 [ 8235.893689] LNet: Removed LNI 192.168.204.115@tcp42 [ 8235.895484] LNet: Removed LNI 192.168.204.115@tcp43 [ 8235.897224] LNet: Removed LNI 192.168.204.115@tcp44 [ 8235.898905] LNet: Removed LNI 192.168.204.115@tcp45 [ 8235.900710] LNet: Removed LNI 192.168.204.115@tcp46 [ 8235.902410] LNet: Removed LNI 192.168.204.115@tcp47 [ 8235.903955] LNet: Removed LNI 192.168.204.115@tcp48 [ 8235.905650] LNet: Removed LNI 192.168.204.115@tcp49 [ 8235.914684] LNet: Removed LNI 192.168.204.115@tcp50 [ 8236.363958] Key type .llcrypt unregistered [ 8236.366221] Key type ._llcrypt unregistered [ 8245.925816] Key type ._llcrypt registered [ 8245.927337] Key type .llcrypt registered [ 8245.985886] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8253.716059] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8254.179731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8254.217908] alg: No test for adler32 (adler32-zlib) [ 8255.136289] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 8255.269576] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8256.880157] Key type lgssc registered [ 8257.513520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8273.756683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.766910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.885411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8274.897352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8274.933528] Lustre: lustre-MDT0000: new disk, initializing [ 8274.957583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.964720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8276.542378] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8282.545936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.613032] Lustre: 485892:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8282.658586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8282.661092] Lustre: Skipped 1 previous similar message [ 8282.703985] Lustre: lustre-MDT0001: new disk, initializing [ 8282.752216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8282.767993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8282.773088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8284.583366] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8288.060572] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8289.270462] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8292.103542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8292.192418] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8292.227097] Lustre: lustre-OST0000: new disk, initializing [ 8292.229661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8292.233926] Lustre: 487780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8292.270523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.503559] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8298.608974] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8301.533467] Lustre: server umount lustre-OST0000 complete [ 8303.585711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8303.589847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8303.596935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.681547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.682326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.298799] Lustre: server umount lustre-MDT0000 complete [ 8310.576550] LustreError: 485887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035871 with bad export cookie 5035795508275466017 [ 8310.579881] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8310.580561] LustreError: 485887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8310.695292] Lustre: server umount lustre-MDT0001 complete [ 8314.070885] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8315.041098] Key type lgssc unregistered [ 8315.174460] LNet: 489598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8315.177619] LNetError: 489598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.184337] LNet: Removed LNI 192.168.204.115@tcp [ 8315.488162] Key type .llcrypt unregistered [ 8315.489326] Key type ._llcrypt unregistered [ 8323.079153] Key type ._llcrypt registered [ 8323.080210] Key type .llcrypt registered [ 8323.118117] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8323.458772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8323.501822] alg: No test for adler32 (adler32-zlib) [ 8324.355547] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 8324.439309] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8326.024118] Key type lgssc registered [ 8326.381748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8328.932375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8328.940040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8330.123478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.416535] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8334.736582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.876042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8336.182964] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8338.261752] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8339.419061] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8341.659496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8341.790362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.694135] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8344.874565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8344.879313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8345.914381] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8356.160669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8359.929985] loop0: detected capacity change from 0 to 204800000 [ 8360.053835] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8360.057493] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8360.061434] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8362.076217] loop0: detected capacity change from 0 to 204800000 [ 8362.141988] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8362.145210] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8362.149484] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8363.879948] loop0: detected capacity change from 0 to 204800000 [ 8365.685825] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8365.732274] Lustre: fs15246-OST0000: new disk, initializing [ 8365.734520] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8365.737171] Lustre: 494245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8365.759385] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.548316] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8371.747177] loop1: detected capacity change from 0 to 204800000 [ 8373.632143] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.675172] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8373.702811] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8373.738853] Lustre: fs15246-MDT0000: new disk, initializing [ 8373.799248] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.808633] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8373.854495] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8374.881893] Lustre: 495397:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000009e66d0ff x1867551484183040/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781035951 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8375.196312] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8383.478188] LustreError: 495397:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8383.483359] LustreError: 495397:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8383.486419] LustreError: 495397:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8386.531958] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.539789] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.649712] Lustre: server umount fs15246-MDT0000 complete [ 8391.651060] LustreError: 490822:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.656984] LustreError: 490822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8392.750661] Lustre: server umount fs15246-OST0000 complete [ 8396.257989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8396.262730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8396.270709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.890131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.892267] Lustre: Skipped 2 previous similar messages [ 8402.257968] Lustre: server umount lustre-OST0000 complete [ 8406.496922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8406.500366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.504117] Lustre: Skipped 1 previous similar message [ 8406.506029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8409.705625] Lustre: server umount lustre-MDT0000 complete [ 8410.976426] LustreError: 490808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781035971 with bad export cookie 1899471765445293347 [ 8410.981349] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8410.984847] LustreError: 490808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8411.123692] Lustre: server umount lustre-MDT0001 complete [ 8414.824949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8415.850204] Key type lgssc unregistered [ 8415.979378] LNet: 498503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8415.982804] LNetError: 498503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8415.992308] LNet: Removed LNI 192.168.204.115@tcp [ 8416.276637] Key type .llcrypt unregistered [ 8416.278642] Key type ._llcrypt unregistered [ 8423.661650] Key type ._llcrypt registered [ 8423.662737] Key type .llcrypt registered [ 8423.698444] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8424.160991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8424.218355] alg: No test for adler32 (adler32-zlib) [ 8425.071444] Lustre: Lustre: Build Version: 2.17.54_1_g05c2c2a [ 8425.158222] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8426.736143] Key type lgssc registered [ 8427.137366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8440.436487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8444.795990] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8459.548078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8459.559904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.699547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8460.715595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8460.767240] Lustre: lustre-MDT0000: new disk, initializing [ 8460.801353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.811988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8462.226897] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8468.134210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.200061] Lustre: 502886:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8468.239006] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8468.242938] Lustre: Skipped 1 previous similar message [ 8468.289842] Lustre: lustre-MDT0001: new disk, initializing [ 8468.335566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.353064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8468.358842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.852463] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8473.940972] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8475.795273] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.383516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.534953] Lustre: lustre-OST0000: new disk, initializing [ 8479.538254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8479.543909] Lustre: 504724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8479.592913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.975648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8479.981876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8480.045663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8482.499521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8487.253771] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8489.467721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8489.605735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8491.688440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8491.825206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8495.588636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8495.591209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.594545] LustreError: Skipped 1 previous similar message [ 8495.604307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8495.610228] Lustre: Skipped 1 previous similar message [ 8498.865143] Lustre: server umount lustre-OST0000 complete [ 8504.289274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8504.291542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.294721] Lustre: Skipped 1 previous similar message [ 8504.296652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.826955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.828239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.017300] Lustre: server umount lustre-MDT0000 complete [ 8508.772211] LustreError: 502880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781036069 with bad export cookie 16329972670525870741 [ 8508.780659] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.947562] Lustre: server umount lustre-MDT0001 complete [ 8520.252317] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8525.410348] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8542.211713] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8229 sec ========== 16:15:01 (1781036101) [ 8543.047546] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:15:02 (1781036102) === [ 8544.523566] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:15:04 (1781036104) === [ 8563.803422] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8565.162733] Key type lgssc unregistered [ 8565.296386] LNet: 511258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.300491] LNetError: 511258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.311390] LNet: Removed LNI 192.168.204.115@tcp [ 8565.747327] Key type .llcrypt unregistered [ 8565.749897] Key type ._llcrypt unregistered