[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557191632 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003344] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.007895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021283] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026086] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.032135] *** VALIDATE proc *** [ 0.033351] *** VALIDATE cgroup *** [ 0.034019] *** VALIDATE cgroup2 *** [ 0.035375] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037286] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039041] Spectre V2 : User space: Vulnerable [ 0.040020] Speculative Store Bypass: Vulnerable [ 0.043445] debug: unmapping init [mem 0xffffffffae059000-0xffffffffae060fff] [ 0.046251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048179] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056304] rcu: Hierarchical SRCU implementation. [ 0.058849] smp: Bringing up secondary CPUs ... [ 0.059761] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.066039] smp: Brought up 1 node, 4 CPUs [ 0.068009] smpboot: Max logical packages: 1 [ 0.069009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121301] node 0 deferred pages initialised in 50ms [ 0.126196] devtmpfs: initialized [ 0.128854] x86/mm: Memory block size: 128MB [ 0.133503] gcov: version magic: 0x41383552 [ 0.135350] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143446] pinctrl core: initialized pinctrl subsystem [ 0.144546] [ 0.146008] ************************************************************* [ 0.148011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151010] ** ** [ 0.153012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156015] ** ** [ 0.158010] ** This means that this kernel is built to expose internal ** [ 0.160010] ** IOMMU data structures, which may compromise security on ** [ 0.163012] ** your system. ** [ 0.165009] ** ** [ 0.168009] ** If you see this message and you are not debugging the ** [ 0.170014] ** kernel, report this immediately to your vendor! ** [ 0.172011] ** ** [ 0.174009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177044] ************************************************************* [ 0.180551] NET: Registered protocol family 16 [ 0.182522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188178] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192018] cpuidle: using governor menu [ 0.194268] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196368] PCI: Using configuration type 1 for base access [ 0.199392] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217234] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220397] cryptd: max_cpu_qlen set to 1000 [ 0.223000] ACPI: Added _OSI(Module Device) [ 0.224023] ACPI: Added _OSI(Processor Device) [ 0.227024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240083] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248109] ACPI: Interpreter enabled [ 0.249048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251010] ACPI: Using IOAPIC for interrupt routing [ 0.252090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271688] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290567] acpiphp: Slot [2] registered [ 0.292090] acpiphp: Slot [5] registered [ 0.294089] acpiphp: Slot [6] registered [ 0.295099] acpiphp: Slot [7] registered [ 0.297208] acpiphp: Slot [8] registered [ 0.298155] acpiphp: Slot [9] registered [ 0.300207] acpiphp: Slot [10] registered [ 0.302147] acpiphp: Slot [3] registered [ 0.303107] acpiphp: Slot [4] registered [ 0.305112] acpiphp: Slot [11] registered [ 0.306129] acpiphp: Slot [12] registered [ 0.308117] acpiphp: Slot [13] registered [ 0.310076] acpiphp: Slot [14] registered [ 0.311151] acpiphp: Slot [15] registered [ 0.312095] acpiphp: Slot [16] registered [ 0.313096] acpiphp: Slot [17] registered [ 0.315093] acpiphp: Slot [18] registered [ 0.317119] acpiphp: Slot [19] registered [ 0.318077] acpiphp: Slot [20] registered [ 0.320103] acpiphp: Slot [21] registered [ 0.322099] acpiphp: Slot [22] registered [ 0.324073] acpiphp: Slot [23] registered [ 0.326068] acpiphp: Slot [24] registered [ 0.327082] acpiphp: Slot [25] registered [ 0.328079] acpiphp: Slot [26] registered [ 0.330092] acpiphp: Slot [27] registered [ 0.332097] acpiphp: Slot [28] registered [ 0.334119] acpiphp: Slot [29] registered [ 0.337197] acpiphp: Slot [30] registered [ 0.338055] acpiphp: Slot [31] registered [ 0.339000] PCI host bridge to bus 0000:00 [ 0.339000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358333] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.383050] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396669] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416656] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422082] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.424067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.427257] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.433012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.447015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.454795] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461719] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.490016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.519018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.528013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.559033] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.576950] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.592018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.614014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.644113] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.662018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.685028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.698638] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.705015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.712014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740875] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.751166] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.766017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.822023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831914] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.835377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.837352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.840547] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.843231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.848099] iommu: Default domain type: Passthrough [ 0.851548] SCSI subsystem initialized [ 0.853187] ACPI: bus type USB registered [ 0.855132] usbcore: registered new interface driver usbfs [ 0.859087] usbcore: registered new interface driver hub [ 0.860074] usbcore: registered new device driver usb [ 0.862149] pps_core: LinuxPPS API ver. 1 registered [ 0.863008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.866055] PTP clock support registered [ 0.868143] EDAC MC: Ver: 3.0.0 [ 0.870264] PCI: Using ACPI for IRQ routing [ 0.872724] NetLabel: Initializing [ 0.874007] NetLabel: domain hash size = 128 [ 0.875007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.876092] NetLabel: unlabeled traffic allowed by default [ 0.879203] vgaarb: loaded [ 0.882150] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.885009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.892212] clocksource: Switched to clocksource kvm-clock [ 1.082498] VFS: Disk quotas dquot_6.6.0 [ 1.084401] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.088336] *** VALIDATE ramfs *** [ 1.090197] *** VALIDATE hugetlbfs *** [ 1.092964] pnp: PnP ACPI init [ 1.096125] pnp: PnP ACPI: found 6 devices [ 1.114714] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.119128] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.122079] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.124803] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.127428] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.130517] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.133534] NET: Registered protocol family 2 [ 1.136991] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.143753] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.147321] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.153058] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.157955] TCP: Hash tables configured (established 65536 bind 65536) [ 1.162162] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.166537] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.170962] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.174457] NET: Registered protocol family 1 [ 1.177881] RPC: Registered named UNIX socket transport module. [ 1.180259] RPC: Registered udp transport module. [ 1.182299] RPC: Registered tcp transport module. [ 1.184011] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.186492] NET: Registered protocol family 44 [ 1.189141] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.193033] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.195533] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.198536] PCI: CLS 0 bytes, default 64 [ 1.200632] Unpacking initramfs... [ 2.804503] debug: unmapping init [mem 0xffff8b073cc54000-0xffff8b073ffbffff] [ 2.809121] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.811763] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.815118] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.462639] Initialise system trusted keyrings [ 3.464750] Key type blacklist registered [ 3.469047] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.505157] zbud: loaded [ 3.514371] *** VALIDATE nfs *** [ 3.540599] *** VALIDATE nfs4 *** [ 3.543176] pstore: using deflate compression [ 3.550185] Platform Keyring initialized [ 3.725500] NET: Registered protocol family 38 [ 3.729996] Key type asymmetric registered [ 3.735309] Asymmetric key parser 'x509' registered [ 3.741496] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.751649] io scheduler mq-deadline registered [ 3.753155] io scheduler kyber registered [ 3.757200] io scheduler bfq registered [ 3.763260] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.770516] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.780808] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.959658] ACPI: Power Button [PWRF] [ 3.965907] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.972962] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.997907] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.011023] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.060431] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.089457] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.118732] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.123608] Non-volatile memory driver v1.3 [ 4.125599] Linux agpgart interface v0.103 [ 4.166645] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.169750] vda: detected capacity change from 0 to 69001216 [ 4.186088] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.189392] vdb: detected capacity change from 0 to 1073741824 [ 4.206879] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.210796] vdc: detected capacity change from 0 to 2621440000 [ 4.237304] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.241159] vdd: detected capacity change from 0 to 2621440000 [ 4.312731] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.317865] vde: detected capacity change from 0 to 4294967296 [ 4.657356] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.666869] vdf: detected capacity change from 0 to 4294967296 [ 4.753169] libphy: Fixed MDIO Bus: probed [ 4.758404] usbcore: registered new interface driver usbserial_generic [ 4.761042] usbserial: USB Serial support registered for generic [ 4.763805] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.768220] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.770295] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.772819] mousedev: PS/2 mouse device common for all mice [ 4.775707] rtc_cmos 00:05: RTC can wake from S4 [ 4.780036] rtc_cmos 00:05: registered as rtc0 [ 4.782165] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.784692] intel_pstate: CPU model not supported [ 4.787383] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.794935] hid: raw HID events driver (C) Jiri Kosina [ 4.798574] usbcore: registered new interface driver usbhid [ 4.800930] usbhid: USB HID core driver [ 4.802459] drop_monitor: Initializing network drop monitor service [ 4.805159] Initializing XFRM netlink socket [ 4.807193] NET: Registered protocol family 10 [ 4.810393] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.813943] Segment Routing with IPv6 [ 4.816732] NET: Registered protocol family 17 [ 4.819584] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.823650] mpls_gso: MPLS GSO support [ 4.830264] RAS: Correctable Errors collector initialized. [ 4.832757] AVX version of gcm_enc/dec engaged. [ 4.834647] AES CTR mode by8 optimization enabled [ 4.916424] sched_clock: Marking stable (4916343817, 0)->(6007023442, -1090679625) [ 4.920949] registered taskstats version 1 [ 4.923311] Loading compiled-in X.509 certificates [ 4.925910] zswap: loaded using pool lzo/zbud [ 4.953480] Key type big_key registered [ 4.965223] Key type encrypted registered [ 4.967050] ima: No TPM chip found, activating TPM-bypass! [ 4.969397] ima: Allocated hash algorithm: sha1 [ 4.970906] ima: No architecture policies found [ 4.972498] evm: Initialising EVM extended attributes: [ 4.974492] evm: security.selinux [ 4.975772] evm: security.ima [ 4.976829] evm: security.capability [ 4.978286] evm: HMAC attrs: 0x1 [ 4.980569] rtc_cmos 00:05: setting system clock to 2026-05-27 08:13:16 UTC (1779869596) [ 4.990763] debug: unmapping init [mem 0xffffffffaf003000-0xffffffffaf1fffff] [ 4.994987] debug: unmapping init [mem 0xffffffffadd82000-0xffffffffae058fff] [ 5.002205] Write protecting the kernel read-only data: 28672k [ 5.005801] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 5.008940] debug: unmapping init [mem 0xffffffffacd14000-0xffffffffacdfffff] [ 5.047721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.057369] systemd[1]: Detected virtualization kvm. [ 5.059824] systemd[1]: Detected architecture x86-64. [ 5.061830] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.086600] systemd[1]: No hostname configured. [ 5.087957] systemd[1]: Set hostname to . [ 5.089766] random: systemd: uninitialized urandom read (16 bytes read) [ 5.092272] systemd[1]: Initializing machine ID from random generator. [ 5.200473] random: ln: uninitialized urandom read (6 bytes read) [ 5.429869] random: systemd: uninitialized urandom read (16 bytes read) [ 5.431842] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.440803] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.449342] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK [ 5.923390] random: fast init done ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.520494] device-mapper: uevent: version 1.0.3 [ 6.524235] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.680413] virtio_net virtio0 ens2: renamed from eth0 [ 8.016883] scsi host0: ata_piix [ 8.071763] scsi host1: ata_piix [ 8.075129] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.079804] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.625851] random: crng init done [ 13.627400] random: 7 urandom warning(s) missed due to ratelimiting [ 16.082199] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.857582] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.664986] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.056982] SELinux: Disabled at runtime. [ 20.159086] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.171428] systemd[1]: Detected virtualization kvm. [ 20.175060] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.181171] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.190023] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.206517] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.219202] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.225079] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.239248] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.264930] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ 21.537048] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 22.995524] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.063252] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.326487] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.116192] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.209963] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.776902] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.499678] NFS: Registering the id_resolver key type [ 29.503761] Key type id_resolver registered [ 29.506552] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 68.878327] hrtimer: interrupt took 5000128 ns [ 100.703850] libcfs: loading out-of-tree module taints kernel. [ 100.728444] Key type ._llcrypt registered [ 100.730415] Key type .llcrypt registered [ 100.833300] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 115.871695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 116.809300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.819304] alg: No test for adler32 (adler32-zlib) [ 118.075801] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 118.838202] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 120.599159] Key type lgssc registered [ 121.942742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.660501] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.678805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 176.062933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.116424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.385900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.419228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.520579] Lustre: lustre-MDT0000: new disk, initializing [ 177.589782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.610713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.892679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.824290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.892071] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.912253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 191.916325] Lustre: Skipped 1 previous similar message [ 191.965826] Lustre: lustre-MDT0001: new disk, initializing [ 192.029795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.055195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.060164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.308674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.420596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.313879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.539328] Lustre: lustre-OST0000: new disk, initializing [ 207.542542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.547964] Lustre: 8410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.598219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.744789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.760152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.869583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.031729] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.785355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.914397] Lustre: lustre-OST0001: new disk, initializing [ 225.919225] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.924973] Lustre: 9465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.027645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.491291] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.508171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.518889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.588862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.512610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.001119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.398377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.140893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 259.334711] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 261.147398] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 262.768459] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 263.897985] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed May 27 04:17:35 EDT 2026 [ 277.078434] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 278.132446] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.856244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.860927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.866045] Lustre: Skipped 3 previous similar messages [ 293.870696] Lustre: Skipped 2 previous similar messages [ 298.591670] Lustre: server umount lustre-MDT0000 complete [ 300.960278] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779869892 with bad export cookie 17599143336611509296 [ 300.961626] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.968557] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 304.096356] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.096903] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.098458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.098465] Lustre: Skipped 1 previous similar message [ 304.115370] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 304.130257] Lustre: Skipped 1 previous similar message [ 305.120523] LustreError: 8417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.120640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.137422] Lustre: Skipped 2 previous similar messages [ 310.240476] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.240686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.251730] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 310.268291] Lustre: Skipped 1 previous similar message [ 315.359209] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.517489] Lustre: server umount lustre-MDT0001 complete [ 317.996869] Lustre: server umount lustre-OST0000 complete [ 320.489613] Lustre: server umount lustre-OST0001 complete [ 324.009113] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 330.504055] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 358.237187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.456772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 358.484448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 358.549866] Lustre: lustre-MDT0000: new disk, initializing [ 358.598760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.611672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.877334] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 369.651938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.762385] Lustre: 14977:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 369.841810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 369.852093] Lustre: Skipped 1 previous similar message [ 369.926919] Lustre: lustre-MDT0001: new disk, initializing [ 370.024828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 370.064776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 370.078043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.509720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 379.210294] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.144823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.107861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 387.293315] Lustre: lustre-OST0000: new disk, initializing [ 387.299623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 387.305423] Lustre: 16717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 387.352405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.863958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 388.876793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 388.975397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 392.521573] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 399.270502] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 403.161917] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 403.329770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 407.118324] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 407.294135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 409.573076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.573081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 409.574111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 409.581704] Lustre: Skipped 1 previous similar message [ 409.593974] Lustre: Skipped 3 previous similar messages [ 419.815086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.820354] Lustre: Skipped 3 previous similar messages [ 423.391107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 423.492449] Lustre: server umount lustre-OST0000 complete [ 426.466054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 426.473066] LustreError: Skipped 1 previous similar message [ 426.479183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.800803] Lustre: server umount lustre-MDT0000 complete [ 434.220228] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870025 with bad export cookie 17599143336611510829 [ 434.226469] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 434.230233] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 434.462776] Lustre: server umount lustre-MDT0001 complete [ 437.129255] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:20:28 (1779870028) [ 438.516623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 439.408989] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:20:30 (1779870030) [ 442.661731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.900857] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.908109] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 442.943656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.066854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 447.968537] LustreError: 18977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.724032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.281845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 455.977851] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.784535] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 463.566832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.226980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 470.942827] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 480.226846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.228191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.228860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.238295] Lustre: Skipped 2 previous similar messages [ 480.247252] Lustre: Skipped 3 previous similar messages [ 481.846852] Lustre: server umount lustre-OST0000 complete [ 490.466857] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.480033] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 490.521475] Lustre: server umount lustre-MDT0000 complete [ 492.832982] LustreError: 18958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870084 with bad export cookie 17599143336611511711 [ 492.833845] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.841219] LustreError: 18958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 492.999219] Lustre: server umount lustre-MDT0001 complete [ 499.737921] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 501.947939] Key type lgssc unregistered [ 502.178189] LNet: 22147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.186613] LNetError: 22147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.197976] LNet: Removed LNI 192.168.203.155@tcp [ 502.829706] Key type .llcrypt unregistered [ 502.831127] Key type ._llcrypt unregistered [ 518.588070] Key type ._llcrypt registered [ 518.589939] Key type .llcrypt registered [ 518.672236] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 519.177254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 519.196744] alg: No test for adler32 (adler32-zlib) [ 520.172607] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 520.335986] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 521.983184] Key type lgssc registered [ 522.727631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 528.638098] loop: module loaded [ 528.684215] loop0: detected capacity change from 0 to 204800000 [ 528.814724] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 528.825348] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 529.037369] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.198050] loop0: detected capacity change from 0 to 204800000 [ 533.307402] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.314855] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.322507] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 537.054793] loop0: detected capacity change from 0 to 204800000 [ 540.998295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 541.017442] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.187296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 542.230462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 542.294791] Lustre: lustre-MDT0000: new disk, initializing [ 542.389184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.410287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 545.331071] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 552.159422] loop1: detected capacity change from 0 to 204800000 [ 556.627717] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.831900] Lustre: lustre-OST07c6: new disk, initializing [ 556.834589] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 556.837175] Lustre: Skipped 1 previous similar message [ 556.842306] Lustre: 25274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 556.887132] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 560.046609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 560.052679] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 560.083557] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 561.123114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 564.821236] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 564.827032] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 564.831713] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 564.838653] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 564.845785] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 564.851860] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 577.628751] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 577.637388] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 577.645890] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 577.651629] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 577.657587] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 577.662459] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 581.800201] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 581.809622] Lustre: 24279:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 581.814868] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 581.822920] Lustre: 24279:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 581.827270] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 581.831669] Lustre: 24279:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 581.838471] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 581.844337] Lustre: 24279:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 581.848796] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 581.853859] Lustre: 24279:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 581.857452] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 581.863270] Lustre: 24279:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 584.992566] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 585.000991] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 585.005626] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 585.010563] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.015304] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 585.022314] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.027323] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 585.032906] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.037438] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 585.041447] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.045588] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.050034] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 592.018824] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 592.028390] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 592.037800] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 592.048038] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 592.056735] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 592.061868] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 592.066532] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 592.071071] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 592.075181] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 592.078266] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 592.081554] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 592.085366] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 602.077807] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 602.088103] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 602.096780] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 602.101323] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 602.107209] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 602.113071] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 602.117428] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 602.122580] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 602.128191] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 602.133108] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 602.139605] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.144219] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 647.136642] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 647.145251] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.153097] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 651.255594] Lustre: server umount lustre-OST07c6 complete [ 657.354601] Lustre: server umount lustre-MDT0000 complete [ 666.972869] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 668.562133] Key type lgssc unregistered [ 668.768288] LNet: 27820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.773710] LNetError: 27820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.788492] LNet: Removed LNI 192.168.203.155@tcp [ 669.312776] Key type .llcrypt unregistered [ 669.317136] Key type ._llcrypt unregistered [ 681.478274] Key type ._llcrypt registered [ 681.479813] Key type .llcrypt registered [ 681.589528] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 682.232988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.308158] alg: No test for adler32 (adler32-zlib) [ 683.334251] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 683.495270] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 685.151197] Key type lgssc registered [ 685.989322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.602966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.618927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.942178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.401425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 702.690167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.954193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 706.092676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 709.951975] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 712.580672] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 716.927077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.150685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.209478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 720.852237] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 724.131406] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 738.787742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.790514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 738.799036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.809498] Lustre: Skipped 1 previous similar message [ 743.906097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 743.910679] Lustre: Skipped 1 previous similar message [ 744.094154] Lustre: server umount lustre-OST0000 complete [ 749.030251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 749.035066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.037535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.045813] Lustre: Skipped 2 previous similar messages [ 752.676811] Lustre: server umount lustre-MDT0000 complete [ 754.145591] LustreError: 29048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 754.161977] LustreError: 29048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 754.933228] LustreError: 29033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870346 with bad export cookie 15437418608091460440 [ 754.937118] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.940333] LustreError: 29033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 755.157232] Lustre: server umount lustre-MDT0001 complete [ 761.875511] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 763.590190] Key type lgssc unregistered [ 763.815574] LNet: 32426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.821322] LNetError: 32426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.843533] LNet: Removed LNI 192.168.203.155@tcp [ 764.358195] Key type .llcrypt unregistered [ 764.360622] Key type ._llcrypt unregistered [ 776.047558] Key type ._llcrypt registered [ 776.049953] Key type .llcrypt registered [ 776.139826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 776.764750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.786037] alg: No test for adler32 (adler32-zlib) [ 777.742771] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 777.923909] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 779.559157] Key type lgssc registered [ 780.311581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 785.620513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 785.632518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.928496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.647733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 795.314706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.507506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 798.043607] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 801.669712] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.008739] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.400484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.606331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.693793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 812.120800] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 815.470655] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 827.362754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.366553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.375726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.433153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.434175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.439941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.570137] Lustre: server umount lustre-OST0000 complete [ 835.553499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.563922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.570654] Lustre: Skipped 2 previous similar messages [ 840.673671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 840.676418] Lustre: Skipped 1 previous similar message [ 845.792848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 845.797321] Lustre: Skipped 1 previous similar message [ 848.351117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 848.473831] Lustre: server umount lustre-MDT0000 complete [ 850.913058] LustreError: 33671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.924177] LustreError: 33671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 850.976515] LustreError: 33653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870442 with bad export cookie 12116223875060610949 [ 850.979509] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 850.984023] LustreError: 33653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 851.212448] Lustre: server umount lustre-MDT0001 complete [ 857.540187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 859.187171] Key type lgssc unregistered [ 859.379567] LNet: 36848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.389588] LNetError: 36848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.406484] LNet: Removed LNI 192.168.203.155@tcp [ 859.964897] Key type .llcrypt unregistered [ 859.966703] Key type ._llcrypt unregistered [ 875.237947] Key type ._llcrypt registered [ 875.240955] Key type .llcrypt registered [ 875.310386] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 875.885491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.957036] alg: No test for adler32 (adler32-zlib) [ 876.934919] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 877.129675] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 878.752249] Key type lgssc registered [ 879.566906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.585879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.603188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.902804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.631749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 894.048735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.297924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 896.773640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 900.119501] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 902.538210] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.747021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.945464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.617833] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 911.088654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 914.157486] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.648066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.652955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.657316] Lustre: Skipped 2 previous similar messages [ 924.666836] Lustre: Skipped 2 previous similar messages [ 926.795556] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 929.235817] Lustre: server umount lustre-MDT0000 complete [ 929.761645] LustreError: 38934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.770726] LustreError: 38934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 931.350794] LustreError: 38204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870522 with bad export cookie 11410338625290460480 [ 931.354294] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.358654] LustreError: 38204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 931.895468] LustreError: 38220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.915590] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 934.881381] LustreError: 38219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.884739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.909105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 936.929234] LustreError: 38220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.617984] Lustre: server umount lustre-MDT0001 complete [ 956.718766] Lustre: server umount lustre-OST0000 complete [ 965.699018] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 967.500938] Key type lgssc unregistered [ 967.725762] LNet: 41727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.731768] LNetError: 41727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.748933] LNet: Removed LNI 192.168.203.155@tcp [ 968.243849] Key type .llcrypt unregistered [ 968.245410] Key type ._llcrypt unregistered [ 983.074483] Key type ._llcrypt registered [ 983.076555] Key type .llcrypt registered [ 983.190182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 983.763073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.837527] alg: No test for adler32 (adler32-zlib) [ 984.759613] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 984.897368] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 986.503183] Key type lgssc registered [ 987.140738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 992.196409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 992.214635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.546058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.721091] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1002.149630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.357841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.996369] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1008.421179] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1010.610911] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.926645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.147539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.920165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1019.240515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1022.433470] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1032.675231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.678188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.700659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.708908] Lustre: Skipped 1 previous similar message [ 1034.318861] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1036.788590] Lustre: server umount lustre-OST0000 complete [ 1053.152839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.161594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.165627] Lustre: Skipped 2 previous similar messages [ 1053.184634] Lustre: Skipped 1 previous similar message [ 1058.742341] Lustre: server umount lustre-MDT0000 complete [ 1059.297876] LustreError: 45257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.309074] LustreError: 45257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1061.582198] LustreError: 43088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870653 with bad export cookie 9992151436796429718 [ 1061.584036] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.591236] LustreError: 43088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1067.905605] Lustre: server umount lustre-MDT0001 complete [ 1074.403986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1076.727052] Key type lgssc unregistered [ 1076.979278] LNet: 46456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.986776] LNetError: 46456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.009279] LNet: Removed LNI 192.168.203.155@tcp [ 1077.533618] Key type .llcrypt unregistered [ 1077.535940] Key type ._llcrypt unregistered [ 1092.349085] Key type ._llcrypt registered [ 1092.354733] Key type .llcrypt registered [ 1092.436285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1093.086988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.096636] alg: No test for adler32 (adler32-zlib) [ 1094.068989] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1094.229281] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1095.849059] Key type lgssc registered [ 1096.475721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1100.684599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1100.702673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.998838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.413692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1109.762230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.984918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.766315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1116.346371] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1118.439483] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1123.045900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.278259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.094247] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1127.407223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1130.531488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1139.257615] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1141.525617] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1152.807351] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-27 4h32m23s [ 1153.763785] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1155.044062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.046541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.053410] Lustre: Skipped 2 previous similar messages [ 1155.058158] Lustre: Skipped 1 previous similar message [ 1156.132048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.137606] Lustre: Skipped 3 previous similar messages [ 1158.724533] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1162.213336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.220635] Lustre: Skipped 2 previous similar messages [ 1168.353431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.359519] Lustre: Skipped 3 previous similar messages [ 1169.377463] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1169.512236] Lustre: server umount lustre-MDT0000 complete [ 1170.997763] LustreError: 49445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.017741] LustreError: 49445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1171.979773] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1174.497121] LustreError: 47834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.510131] LustreError: 47834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1176.681898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.795263] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.010319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.057191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1178.026635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1180.188512] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1182.539983] Lustre: DEBUG MARKER: Wait for df (36004) ... [ 1183.556891] Lustre: DEBUG MARKER: done [ 1187.296969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.304942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.312050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.324855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.075739] Lustre: server umount lustre-OST0000 complete [ 1198.561281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1198.561357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.576146] Lustre: Skipped 1 previous similar message [ 1200.658461] Lustre: server umount lustre-MDT0000 complete [ 1203.057331] LustreError: 49448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870794 with bad export cookie 3339147382097309533 [ 1203.062676] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.074187] LustreError: 49448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1203.376342] Lustre: server umount lustre-MDT0001 complete [ 1209.982366] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1211.707628] Key type lgssc unregistered [ 1211.918442] LNet: 52383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.926965] LNetError: 52383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.939151] LNet: Removed LNI 192.168.203.155@tcp [ 1212.476216] Key type .llcrypt unregistered [ 1212.478673] Key type ._llcrypt unregistered [ 1237.464736] Key type ._llcrypt registered [ 1237.466565] Key type .llcrypt registered [ 1237.536717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1238.300413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.323644] alg: No test for adler32 (adler32-zlib) [ 1239.288055] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1239.426728] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1241.031284] Key type lgssc registered [ 1241.678537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1246.680453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1246.693141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.889600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1247.920116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1248.048229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.790681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1256.651244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.731673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1256.759195] Lustre: 54551:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1256.945193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1259.327458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1262.579869] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.567790] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1268.776207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.912602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1269.043731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.542249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1272.591243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1276.236322] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1281.729352] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-27 4h34m33s [ 1282.707476] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1287.415729] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1289.074575] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1289.114386] LustreError: 54558:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1309.261410] Lustre: lustre-MDT0000: Client 03ff6c68-8759-41bd-8614-e5167f7a7f45 (at 192.168.203.55@tcp) reconnecting [ 1329.221829] LustreError: 54558:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1330.778810] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1332.779151] Lustre: DEBUG MARKER: done [ 1338.336590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.343376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.346299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.352809] Lustre: Skipped 1 previous similar message [ 1338.368197] Lustre: Skipped 1 previous similar message [ 1342.723191] Lustre: server umount lustre-OST0000 complete [ 1348.577581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.582936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.587220] Lustre: Skipped 1 previous similar message [ 1349.664959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.668612] Lustre: Skipped 2 previous similar messages [ 1351.590689] Lustre: server umount lustre-MDT0000 complete [ 1354.029747] LustreError: 56184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870945 with bad export cookie 6246507910821154505 [ 1354.030828] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.041159] LustreError: 56184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1354.272301] Lustre: server umount lustre-MDT0001 complete [ 1361.217380] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1363.219546] Key type lgssc unregistered [ 1363.490900] LNet: 58518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.495880] LNetError: 58518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.514097] LNet: Removed LNI 192.168.203.155@tcp [ 1364.061339] Key type .llcrypt unregistered [ 1364.063801] Key type ._llcrypt unregistered [ 1395.190344] Key type ._llcrypt registered [ 1395.194086] Key type .llcrypt registered [ 1395.265703] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1396.027987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.040838] alg: No test for adler32 (adler32-zlib) [ 1397.010979] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1397.221757] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1398.887805] Key type lgssc registered [ 1399.754688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1402.114778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1402.136583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.366045] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1403.396237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1403.557692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.269464] Lustre: Failing over lustre-MDT0000 [ 1404.497208] Lustre: server umount lustre-MDT0000 complete [ 1409.308645] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:36:40 (1779871000) [ 1413.932407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.220781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.653410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1422.232054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.304800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1422.335778] Lustre: 61276:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1422.528223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.981929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1428.150089] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.292511] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1434.433321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.596175] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1434.729487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.304942] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1441.809767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1442.335817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1447.079810] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1452.515658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.519210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.525655] Lustre: Skipped 2 previous similar messages [ 1452.537066] Lustre: Skipped 2 previous similar messages [ 1453.604605] Lustre: server umount lustre-MDT0000 complete [ 1456.114921] LustreError: 61995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871047 with bad export cookie 4922012065758270592 [ 1456.117745] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.122970] LustreError: 61995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1456.306261] Lustre: server umount lustre-MDT0001 complete [ 1458.282796] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1461.229973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1466.539896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.759398] Lustre: 60205:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871049/real 1779871049] req@0000000041950c36 x1866330001146880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1473.785290] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.047158] Lustre: 60203:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871053/real 1779871053] req@00000000df42691e x1866330001147008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1481.254452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.322177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1482.271140] Lustre: 60204:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871057/real 1779871057] req@000000000572366e x1866330001147392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1482.283618] Lustre: 60204:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1483.775729] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1487.391646] Lustre: 60204:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871062/real 1779871062] req@0000000086127306 x1866330001147520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1489.300227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.377366] LustreError: 64706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.396404] LustreError: 64706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1489.537314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1492.487291] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1492.653422] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1496.027170] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1498.597332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1503.205920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.209621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.213821] Lustre: Skipped 1 previous similar message [ 1503.222284] Lustre: Skipped 2 previous similar messages [ 1505.365214] Lustre: server umount lustre-MDT0000 complete [ 1507.753280] LustreError: 61269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871099 with bad export cookie 4922012065758272867 [ 1507.756574] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.762432] LustreError: 61269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1507.923391] Lustre: server umount lustre-MDT0001 complete [ 1512.168470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1518.511189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.679250] Lustre: 60205:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871099/real 1779871099] req@00000000734d55d9 x1866330001168384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1523.708448] Lustre: 60205:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1523.712541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.721724] Lustre: Skipped 1 previous similar message [ 1532.959693] LustreError: 60201:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ac289377 x1866330001169408/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1533.120925] LustreError: 62899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.164878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.220212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1533.919325] Lustre: 60205:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871109/real 1779871109] req@0000000085a86871 x1866330001168896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779871125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1533.936937] Lustre: 60205:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1534.177582] LustreError: 67159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.189369] LustreError: 67159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1535.831247] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1539.555965] LustreError: 67158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.568424] LustreError: 67158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1541.208583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.462223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1543.467967] Lustre: Skipped 1 previous similar message [ 1543.857482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1547.131545] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.511985] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1553.893551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.900738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.901490] Lustre: Skipped 1 previous similar message [ 1553.911971] Lustre: Skipped 2 previous similar messages [ 1559.018411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.030355] Lustre: Skipped 2 previous similar messages [ 1559.178839] Lustre: server umount lustre-MDT0000 complete [ 1562.023244] LustreError: 61268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871153 with bad export cookie 4922012065758274246 [ 1562.025273] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1562.030394] LustreError: 61268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1562.208597] Lustre: server umount lustre-MDT0001 complete [ 1566.797973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1569.127087] Lustre: DEBUG MARKER: files compared the same [ 1579.777268] Lustre: server umount lustre-OST0000 complete [ 1590.086374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1591.738038] Key type lgssc unregistered [ 1591.945728] LNet: 70190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.953545] LNetError: 70190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.966940] LNet: Removed LNI 192.168.203.155@tcp [ 1592.440560] Key type .llcrypt unregistered [ 1592.442680] Key type ._llcrypt unregistered [ 1608.076070] Key type ._llcrypt registered [ 1608.081040] Key type .llcrypt registered [ 1608.174156] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1608.789767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.856086] alg: No test for adler32 (adler32-zlib) [ 1609.820260] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1609.985936] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1611.632786] Key type lgssc registered [ 1612.572280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1618.552599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1618.577523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.970540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.480327] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 1628.132755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.359247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.084625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 1634.966761] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1637.503704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1641.910415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.135807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.221832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1645.912128] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 1649.691558] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1660.896515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1660.901915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.911254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.971172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.971806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.979627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.805848] Lustre: server umount lustre-OST0000 complete [ 1669.091604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.099890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.102907] Lustre: Skipped 1 previous similar message [ 1669.111292] Lustre: Skipped 2 previous similar messages [ 1674.214913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.379168] Lustre: server umount lustre-MDT0000 complete [ 1676.356848] LustreError: 73193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871267 with bad export cookie 7086269251328436589 [ 1676.361262] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.364388] LustreError: 73193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1676.545785] Lustre: server umount lustre-MDT0001 complete [ 1682.201714] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1683.807572] Key type lgssc unregistered [ 1684.018296] LNet: 74751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.024221] LNetError: 74751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.038325] LNet: Removed LNI 192.168.203.155@tcp [ 1684.579190] Key type .llcrypt unregistered [ 1684.581928] Key type ._llcrypt unregistered [ 1698.498681] Key type ._llcrypt registered [ 1698.500671] Key type .llcrypt registered [ 1698.563205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1699.011649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.035063] alg: No test for adler32 (adler32-zlib) [ 1699.947323] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1700.074339] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1701.687157] Key type lgssc registered [ 1702.275687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1706.550241] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1706.560813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1722.080929] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1727.199250] LustreError: 76103:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 1753.823194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.403287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1760.810442] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1765.069699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.154384] Lustre: *** cfs_fail_loc=706, val=0*** [ 1765.348296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.388799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1767.408085] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1772.132400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.370283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1774.563336] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1777.602052] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1779.556540] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1780.704069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.707894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.722871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.753245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.757903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.758217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.763885] Lustre: Skipped 1 previous similar message [ 1786.798487] Lustre: server umount lustre-OST0000 complete [ 1792.993684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.994276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1792.995597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.003796] Lustre: Skipped 1 previous similar message [ 1798.112757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.119187] Lustre: Skipped 2 previous similar messages [ 1803.231472] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1803.232391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.241154] Lustre: Skipped 1 previous similar message [ 1803.312610] Lustre: server umount lustre-MDT0000 complete [ 1805.498182] LustreError: 76111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871397 with bad export cookie 18028320525555019448 [ 1805.500208] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.504854] LustreError: 76111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1805.662328] Lustre: server umount lustre-MDT0001 complete [ 1814.128192] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1817.609368] Key type lgssc unregistered [ 1818.041896] LNet: 79360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.064397] LNetError: 79360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.097986] LNet: Removed LNI 192.168.203.155@tcp [ 1818.888378] Key type .llcrypt unregistered [ 1818.892237] Key type ._llcrypt unregistered [ 1853.350695] Key type ._llcrypt registered [ 1853.354974] Key type .llcrypt registered [ 1853.517298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1855.414611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.443430] alg: No test for adler32 (adler32-zlib) [ 1857.043804] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1857.610575] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1859.423162] Key type lgssc registered [ 1861.366467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1870.982122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.019662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.216545] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1886.231843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.881511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.943388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1892.324589] LustreError: 81430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.341456] LustreError: 81430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1897.461992] LustreError: 81434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.484258] LustreError: 81434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1901.064107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1901.536280] LustreError: 81430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.569500] LustreError: 81430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1901.592225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.657577] LustreError: 81433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.672402] LustreError: 81433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1908.260763] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1911.780834] LustreError: 82161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.814224] LustreError: 82161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1917.352619] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.361772] Lustre: MGS: Client 978474f6-9cab-4354-8a21-8dddfeae7ab8 (at 0@lo) reconnecting [ 1917.391786] LustreError: 81444:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1917.642170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.720199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1921.378259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1938.405121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1938.408916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.425604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.440190] Lustre: Skipped 1 previous similar message [ 1938.469841] Lustre: Skipped 1 previous similar message [ 1939.973395] Lustre: server umount lustre-OST0000 complete [ 1948.640721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.656496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.661110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.686288] Lustre: Skipped 1 previous similar message [ 1951.973301] Lustre: server umount lustre-MDT0000 complete [ 1953.764846] LustreError: 81433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.781751] LustreError: 81433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1956.137414] Lustre: server umount lustre-MDT0001 complete [ 1960.144344] Lustre: server umount MGS complete [ 1972.617229] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:46:03 (1779871563) [ 2002.828804] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2011.542651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2044.198771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.023983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2057.842545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.144621] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2058.197906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2058.294606] Lustre: lustre-MDT0001: new disk, initializing [ 2058.382967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.439490] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2058.452572] Lustre: 88518:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2058.474757] LustreError: 88518:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2063.350131] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2074.460827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.695629] Lustre: lustre-OST0000: new disk, initializing [ 2074.699270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2074.704842] Lustre: 89268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2074.782497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.640113] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2085.476441] Lustre: 89762:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000000f78bdd6 x1866330484547456/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779871692 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2088.702257] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2097.472715] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.482657] Lustre: MGS: Client c91e3768-eaf5-4082-9c47-709c10aa95c9 (at 0@lo) reconnecting [ 2097.522902] Lustre: 87805:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2097.592874] LustreError: 88504:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2097.606988] LustreError: 88504:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2097.638376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2097.870114] Lustre: lustre-MDT0000: new disk, initializing [ 2098.093287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.129977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2098.227507] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2099.168883] Lustre: 90224:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008fd0199a x1866330484554368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1779871706 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2103.326413] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2104.805095] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2108.330487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2108.347469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2108.480671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2134.499190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2134.507814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.523476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.528929] Lustre: Skipped 1 previous similar message [ 2137.569077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.580377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.602496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.654229] Lustre: server umount lustre-OST0000 complete [ 2148.321877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.324378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.347652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.347661] Lustre: Skipped 2 previous similar messages [ 2150.131132] Lustre: server umount lustre-MDT0000 complete [ 2153.440661] LustreError: 88493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.464225] LustreError: 88493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2160.871098] Lustre: server umount lustre-MDT0001 complete [ 2164.917504] Lustre: server umount MGS complete [ 2176.504434] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:49:27 (1779871767) [ 2195.299183] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2198.411961] Key type lgssc unregistered [ 2198.717805] LNet: 92692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.724933] LNetError: 92692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.781378] LNet: Removed LNI 192.168.203.155@tcp [ 2200.517629] Key type .llcrypt unregistered [ 2200.520070] Key type ._llcrypt unregistered [ 2222.908731] Key type ._llcrypt registered [ 2222.910741] Key type .llcrypt registered [ 2223.116935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2224.471943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.500438] alg: No test for adler32 (adler32-zlib) [ 2225.750280] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2225.983315] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2227.695167] Key type lgssc registered [ 2229.001298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2238.260127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2238.314122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.906767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.435400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2250.212502] LustreError: 93900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.231326] LustreError: 93900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.521081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.932094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.401979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2264.922618] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2268.891414] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2275.964610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.313030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.983398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2284.538810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2289.332836] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2299.884196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.886671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.893313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.894688] Lustre: Skipped 1 previous similar message [ 2303.964902] Lustre: server umount lustre-OST0000 complete [ 2310.112073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.112650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.130226] Lustre: Skipped 1 previous similar message [ 2310.139804] Lustre: Skipped 2 previous similar messages [ 2314.577777] Lustre: server umount lustre-MDT0000 complete [ 2315.235057] LustreError: 93897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.249601] LustreError: 93897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2318.746034] LustreError: 93882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871910 with bad export cookie 15273067671755992393 [ 2318.757499] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.766993] LustreError: 93882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.434130] Lustre: server umount lustre-MDT0001 complete [ 2331.367898] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2334.325497] Key type lgssc unregistered [ 2334.760754] LNet: 97076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.775254] LNetError: 97076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.852982] LNet: Removed LNI 192.168.203.155@tcp [ 2336.918354] Key type .llcrypt unregistered [ 2336.926358] Key type ._llcrypt unregistered [ 2357.563585] Key type ._llcrypt registered [ 2357.569265] Key type .llcrypt registered [ 2357.908700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2359.257494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.355546] alg: No test for adler32 (adler32-zlib) [ 2360.747822] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2360.971741] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2362.711804] Key type lgssc registered [ 2364.598486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.951409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2371.959275] LustreError: 98135:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2372.042121] LustreError: 98153:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2372.047677] LustreError: 98135:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4921 [ 2372.054099] LustreError: 98135:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2372.063533] LustreError: 98135:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2372.079775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.661486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.746651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.190504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.148820] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2391.455364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 2394.152389] LustreError: 99323:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2394.185796] LustreError: 99327:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2394.190931] LustreError: 99323:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 2394.198696] LustreError: 99323:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2394.209602] LustreError: 99323:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2394.235179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2394.643728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.899735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.906509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.913854] Lustre: Skipped 1 previous similar message [ 2397.934031] Lustre: Skipped 1 previous similar message [ 2402.788554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2402.796170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.811602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.360378] Lustre: server umount lustre-MDT0000 complete [ 2407.396988] LustreError: 98190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.418816] LustreError: 98190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2408.192726] LustreError: 98172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871999 with bad export cookie 6077196648695921051 [ 2408.200680] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.204413] LustreError: 98172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2409.567332] LustreError: 98621:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002d6b74d2 x1866331011960576/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2409.733320] Lustre: server umount lustre-MDT0001 complete [ 2423.823452] Lustre: server umount lustre-OST0000 complete [ 2432.638718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.129385] LustreError: 100391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.220212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.548701] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2438.634084] LustreError: 100392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.734078] LustreError: 100391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.913251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.225762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.351189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2459.144509] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.558684] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.706445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.849969] LustreError: 101987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.095109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.576826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2478.956393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2485.813040] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.920677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.923312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.948266] Lustre: Skipped 1 previous similar message [ 2493.963931] Lustre: Skipped 4 previous similar messages [ 2499.042061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2499.050341] Lustre: Skipped 1 previous similar message [ 2499.404607] Lustre: server umount lustre-OST0000 complete [ 2504.165406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.192111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.198620] Lustre: Skipped 1 previous similar message [ 2510.129874] Lustre: server umount lustre-MDT0000 complete [ 2513.905523] LustreError: 100371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872105 with bad export cookie 6077196648695921751 [ 2513.908650] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.918592] LustreError: 100371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2514.341346] Lustre: server umount lustre-MDT0001 complete [ 2525.906481] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2529.024324] Key type lgssc unregistered [ 2529.306751] LNet: 103567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.328962] LNetError: 103567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.410409] LNet: Removed LNI 192.168.203.155@tcp [ 2531.337755] Key type .llcrypt unregistered [ 2531.346230] Key type ._llcrypt unregistered [ 2555.926478] Key type ._llcrypt registered [ 2555.932217] Key type .llcrypt registered [ 2556.045799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2557.146586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.287848] alg: No test for adler32 (adler32-zlib) [ 2558.466242] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2558.760055] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2560.471166] Key type lgssc registered [ 2561.466865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2569.743930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2569.783383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.316170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.951198] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2581.475319] LustreError: 104972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.498186] LustreError: 104972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2584.998968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.292566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.132299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2596.245303] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2600.716201] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2608.888355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.230898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.700547] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2616.431918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2620.543237] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2626.343101] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2629.518813] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2636.263266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2636.276462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.309686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.281188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.282817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.299294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.322880] Lustre: Skipped 1 previous similar message [ 2639.641966] Lustre: server umount lustre-OST0000 complete [ 2647.009983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2647.022570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.049981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2649.379705] Lustre: server umount lustre-MDT0000 complete [ 2652.652512] LustreError: 104971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.676030] LustreError: 104971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2652.719745] LustreError: 104951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872244 with bad export cookie 15923835632708252352 [ 2652.733633] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2653.110229] Lustre: server umount lustre-MDT0001 complete [ 2665.808337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2668.410526] Key type lgssc unregistered [ 2668.717588] LNet: 108251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2668.744584] LNetError: 108251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2669.797921] LNet: Removed LNI 192.168.203.155@tcp [ 2670.657607] Key type .llcrypt unregistered [ 2670.660788] Key type ._llcrypt unregistered [ 2690.006934] Key type ._llcrypt registered [ 2690.009016] Key type .llcrypt registered [ 2690.111402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2690.750385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.772065] alg: No test for adler32 (adler32-zlib) [ 2691.871827] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2692.061716] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2693.759507] Key type lgssc registered [ 2695.130903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2704.709348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2704.726374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.155984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.808582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2715.616723] LustreError: 109457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.633327] LustreError: 109457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2718.995902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.389994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2723.412066] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2729.096813] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.923621] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2740.458095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.783702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.413624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2751.031243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2755.622280] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2766.310649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.316371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2766.340228] Lustre: Skipped 1 previous similar message [ 2766.340638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.998339] Lustre: server umount lustre-OST0000 complete [ 2775.520133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.529836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.550157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.555198] Lustre: Skipped 1 previous similar message [ 2776.562951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.585377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.597944] Lustre: Skipped 1 previous similar message [ 2780.647926] Lustre: server umount lustre-MDT0000 complete [ 2781.666356] LustreError: 109457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.686587] LustreError: 109457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2784.217913] LustreError: 109438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872375 with bad export cookie 15649553684741474195 [ 2784.226706] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.246463] LustreError: 109438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2784.667099] Lustre: server umount lustre-MDT0001 complete [ 2794.533306] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2797.130894] Key type lgssc unregistered [ 2797.455175] LNet: 112633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.462953] LNetError: 112633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.508028] LNet: Removed LNI 192.168.203.155@tcp [ 2799.281219] Key type .llcrypt unregistered [ 2799.283580] Key type ._llcrypt unregistered [ 2821.683089] Key type ._llcrypt registered [ 2821.684761] Key type .llcrypt registered [ 2821.789764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2822.690766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.776095] alg: No test for adler32 (adler32-zlib) [ 2823.865983] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2824.013350] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2825.663167] Key type lgssc registered [ 2826.513190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2834.273893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.309204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.829783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.285338] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2846.184431] LustreError: 114030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.196056] LustreError: 114030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2848.912544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.361389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.670583] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2858.848914] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2863.758466] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.998453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2872.247111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.764553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2881.459088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2886.296563] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2894.192424] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2898.298247] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2898.309533] Lustre: Skipped 2 previous similar messages [ 2903.368383] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2903.374917] Lustre: Skipped 2 previous similar messages [ 2904.170455] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2904.215674] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2908.652659] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2908.664383] Lustre: Skipped 2 previous similar messages [ 2913.400901] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 2917.086381] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 2917.095519] Lustre: Skipped 2 previous similar messages [ 2918.730816] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 2918.745557] Lustre: Skipped 1 previous similar message [ 2932.705269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.713229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.716200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.731753] Lustre: Skipped 1 previous similar message [ 2932.750649] Lustre: Skipped 1 previous similar message [ 2937.827756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.838017] Lustre: Skipped 1 previous similar message [ 2938.924864] Lustre: server umount lustre-OST0000 complete [ 2946.528349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.545603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.571084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.083509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.415208] Lustre: server umount lustre-MDT0000 complete [ 2953.184420] LustreError: 114030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.201579] LustreError: 114030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2953.391191] LustreError: 116335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872544 with bad export cookie 13636929748202539846 [ 2953.395541] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.408277] LustreError: 116335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2953.939623] Lustre: server umount lustre-MDT0001 complete [ 2965.349448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2968.417581] Key type lgssc unregistered [ 2968.723727] LNet: 118401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2968.737186] LNetError: 118401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2968.758731] LNet: Removed LNI 192.168.203.155@tcp [ 2969.509556] Key type .llcrypt unregistered [ 2969.519880] Key type ._llcrypt unregistered [ 3001.093029] Key type ._llcrypt registered [ 3001.102181] Key type .llcrypt registered [ 3001.288845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3002.140894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3002.217202] alg: No test for adler32 (adler32-zlib) [ 3003.392430] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 3003.562543] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3005.207159] Key type lgssc registered [ 3006.360350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3015.098410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3015.127776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.778535] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3016.817284] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3016.839668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.376080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3027.434627] LustreError: 119947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.452191] LustreError: 119947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3031.162270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.607088] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3031.659900] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3031.767169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.504707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3043.124275] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3047.366824] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3054.983630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.386201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.601044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3062.101790] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3067.844377] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.089477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.091834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.097077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.110326] Lustre: Skipped 1 previous similar message [ 3082.212243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3082.216674] Lustre: Skipped 1 previous similar message [ 3082.809749] Lustre: server umount lustre-OST0000 complete [ 3087.329366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.339812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.346738] Lustre: Skipped 1 previous similar message [ 3087.359391] Lustre: Skipped 1 previous similar message [ 3092.453503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.465176] Lustre: Skipped 1 previous similar message [ 3092.544327] Lustre: server umount lustre-MDT0000 complete [ 3096.092629] LustreError: 119928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872687 with bad export cookie 17880452483779897714 [ 3096.109420] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3096.113273] LustreError: 119928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3096.726250] Lustre: server umount lustre-MDT0001 complete [ 3107.343801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3110.364630] Key type lgssc unregistered [ 3110.738570] LNet: 123176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.743331] LNetError: 123176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3110.756767] LNet: Removed LNI 192.168.203.155@tcp [ 3111.646555] Key type .llcrypt unregistered [ 3111.650514] Key type ._llcrypt unregistered [ 3141.195524] Key type ._llcrypt registered [ 3141.199657] Key type .llcrypt registered [ 3141.361336] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3162.218239] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3163.514941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3163.532688] alg: No test for adler32 (adler32-zlib) [ 3164.669394] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 3164.929595] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3166.631233] Key type lgssc registered [ 3167.548065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3199.441942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3199.462320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.726580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3200.761705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3200.844837] Lustre: lustre-MDT0000: new disk, initializing [ 3200.950837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.980530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3205.123420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3214.662096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.808382] Lustre: 126545:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3214.895609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3214.902218] Lustre: Skipped 1 previous similar message [ 3214.997836] Lustre: lustre-MDT0001: new disk, initializing [ 3215.132151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3215.168601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3215.190459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3218.906660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3227.694609] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.157522] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3239.799587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.000373] Lustre: lustre-OST0000: new disk, initializing [ 3240.003462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3240.008400] Lustre: 128286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3240.093946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.952986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3249.690283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3249.700801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3249.779174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3251.220604] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.590486] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.744887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3259.206224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3259.378374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3265.009780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.011571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3265.011828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3265.022721] Lustre: Skipped 1 previous similar message [ 3265.047970] Lustre: Skipped 1 previous similar message [ 3270.115540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.120289] Lustre: Skipped 1 previous similar message [ 3275.231179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3275.236958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.255302] Lustre: Skipped 1 previous similar message [ 3275.344561] Lustre: server umount lustre-OST0000 complete [ 3280.353745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.363536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.365672] Lustre: Skipped 1 previous similar message [ 3284.615523] Lustre: server umount lustre-MDT0000 complete [ 3287.596368] LustreError: 126536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872879 with bad export cookie 4084745440902999694 [ 3287.598122] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.603635] LustreError: 126536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3287.838502] Lustre: server umount lustre-MDT0001 complete [ 3294.913282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.282429] LustreError: 130352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.299107] LustreError: 130352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3295.349111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.031625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3300.836988] LustreError: 130353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.933494] LustreError: 130352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.407732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.675482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3311.465381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3316.728970] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3321.172140] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3328.999978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.346926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.059791] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3342.550274] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.623534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3346.972652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3352.916344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3353.268977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3361.977911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.234534] Lustre: lustre-OST0001: new disk, initializing [ 3362.239912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3362.254854] Lustre: 133351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3362.411587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.885475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3370.578637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3370.607393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3370.688073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3376.596396] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3376.786716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3381.526973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3381.754304] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3386.057903] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff935047b33000.ost_server_uuid 50 [ 3387.545683] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff935047b33000.ost_server_uuid in FULL state after 0 sec [ 3396.585346] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.585620] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.595962] LustreError: Skipped 1 previous similar message [ 3396.631154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3396.641353] Lustre: Skipped 2 previous similar messages [ 3402.102225] Lustre: server umount lustre-OST0001 complete [ 3406.819217] LustreError: 132420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.820807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.878996] LustreError: 132420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3406.887014] Lustre: Skipped 1 previous similar message [ 3406.889822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.925952] Lustre: Skipped 2 previous similar messages [ 3411.432661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3411.447578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.945977] LustreError: 131948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.996416] LustreError: 131948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3412.664032] Lustre: server umount lustre-OST0000 complete [ 3427.299842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.309400] Lustre: Skipped 7 previous similar messages [ 3431.392381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3431.511174] Lustre: server umount lustre-MDT0000 complete [ 3432.429855] LustreError: 130348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.454161] LustreError: 130348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3435.302229] LustreError: 131969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873026 with bad export cookie 4084745440903000576 [ 3435.308776] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.322759] LustreError: 131969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3435.886540] Lustre: server umount lustre-MDT0001 complete [ 3455.814947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3458.493657] Key type lgssc unregistered [ 3458.812076] LNet: 136197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3458.823446] LNetError: 136197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3458.838221] LNet: Removed LNI 192.168.203.155@tcp [ 3459.809860] Key type .llcrypt unregistered [ 3459.814756] Key type ._llcrypt unregistered [ 3503.847501] Key type ._llcrypt registered [ 3503.850320] Key type .llcrypt registered [ 3503.945304] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3518.732628] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3520.261925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3520.277033] alg: No test for adler32 (adler32-zlib) [ 3521.325725] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 3521.571245] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3523.270114] Key type lgssc registered [ 3524.386693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3558.988362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3559.023434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.410359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3560.432895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3560.478375] Lustre: lustre-MDT0000: new disk, initializing [ 3560.566802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.605183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3564.289206] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3575.488092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.595402] Lustre: 140332:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3575.681706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3575.689879] Lustre: Skipped 1 previous similar message [ 3575.797554] Lustre: lustre-MDT0001: new disk, initializing [ 3575.914599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3575.953532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3575.961556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3581.406775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3590.060781] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3594.859286] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3602.992292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.468598] Lustre: lustre-OST0000: new disk, initializing [ 3603.477075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3603.487127] Lustre: 142074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3603.618554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.638171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.650945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.745993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3610.383568] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3618.145495] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3622.685454] Lustre: Failing over lustre-OST0000 [ 3624.909409] Lustre: server umount lustre-OST0000 complete [ 3641.311847] Lustre: 138559:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779873216/real 1779873216] req@00000000b8b5e529 x1866332229371904/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779873232 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3641.317768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.352559] Lustre: 138559:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3641.398349] Lustre: Skipped 1 previous similar message [ 3645.951121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.408459] Lustre: 138560:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779873221/real 1779873221] req@0000000045813320 x1866332229372928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779873237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3646.429397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.431022] Lustre: 138560:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3646.452397] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3650.527252] Lustre: 138560:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779873226/real 1779873226] req@000000004f0a8a23 x1866332229374080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779873242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3650.561076] Lustre: 138560:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3651.495910] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3651.712089] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3651.712450] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3652.192526] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3655.839494] Lustre: 138557:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779873231/real 1779873231] req@000000005c4d4a7b x1866332229375360/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779873247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3656.261832] Lustre: Failing over lustre-MDT0000 [ 3656.680869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.690162] Lustre: Skipped 1 previous similar message [ 3656.787514] Lustre: server umount lustre-MDT0000 complete [ 3661.796786] LustreError: 140343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.821896] LustreError: 140343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3666.913456] LustreError: 141153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.929675] LustreError: 141153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3672.033562] LustreError: 140344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.046109] LustreError: 140344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3673.055258] Lustre: 140335:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779873248/real 1779873248] req@000000000944897d x1866332229383552/t0(0) o101->MGC192.168.203.155@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1779873264 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3673.083169] Lustre: 140335:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3673.108765] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3673.135233] Lustre: 140335:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3675.563452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.244709] LustreError: 140340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.258882] LustreError: 140340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3682.369416] LustreError: 140340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.389781] LustreError: 140340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3683.851084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.916188] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3687.481499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3688.975868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3688.995192] Lustre: Skipped 1 previous similar message [ 3689.011061] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3689.101282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3690.001973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3703.277294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.286866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.301252] Lustre: Skipped 1 previous similar message [ 3703.304345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.296597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.305742] Lustre: Skipped 1 previous similar message [ 3707.358828] Lustre: server umount lustre-OST0000 complete [ 3714.528649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3714.531168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.534531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.551805] Lustre: Skipped 2 previous similar messages [ 3719.649245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.655583] Lustre: Skipped 2 previous similar messages [ 3724.770572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.783472] Lustre: Skipped 1 previous similar message [ 3726.303140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3726.517392] Lustre: server umount lustre-MDT0000 complete [ 3728.867279] LustreError: 142681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.889173] LustreError: 142681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3729.483045] LustreError: 140324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873321 with bad export cookie 8422086376098764455 [ 3729.489422] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.496113] LustreError: 140324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3729.898368] Lustre: server umount lustre-MDT0001 complete [ 3740.242480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3742.913441] Key type lgssc unregistered [ 3743.202672] LNet: 145569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3743.217349] LNetError: 145569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3743.261278] LNet: Removed LNI 192.168.203.155@tcp [ 3744.131549] Key type .llcrypt unregistered [ 3744.134292] Key type ._llcrypt unregistered [ 3787.648938] Key type ._llcrypt registered [ 3787.654434] Key type .llcrypt registered [ 3787.815565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3789.161109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.211837] alg: No test for adler32 (adler32-zlib) [ 3790.335964] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 3790.534790] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3792.167922] Key type lgssc registered [ 3793.295519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3801.827843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3801.853501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.859483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.377381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3814.393975] LustreError: 147074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.432333] LustreError: 147074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3819.495038] LustreError: 147075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.062402] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3824.613157] LustreError: 147074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.066872] LustreError: 147103:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3826.078184] LustreError: 147103:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3826.085708] LustreError: 147103:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3826.881445] Lustre: server umount lustre-MDT0000 complete [ 3843.828620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3855.748252] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3898.466985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.615974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3898.635865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3898.736545] Lustre: lustre-MDT0000: new disk, initializing [ 3898.825188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.843523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3902.802242] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3913.244784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.390365] Lustre: 150262:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3913.463844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3913.468542] Lustre: Skipped 1 previous similar message [ 3913.541285] Lustre: lustre-MDT0001: new disk, initializing [ 3913.652143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3913.705448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3913.724151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3917.312517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3925.042949] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3929.703838] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3937.042596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.250881] Lustre: lustre-OST0000: new disk, initializing [ 3937.253637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3937.257809] Lustre: 152002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3937.391845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.054224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3946.577317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3946.585737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3946.764683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3953.551332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3959.552114] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3961.400275] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:19:12 (1779873552) [ 4213.215757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.219466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.228206] Lustre: Skipped 1 previous similar message [ 4218.336854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.341935] Lustre: Skipped 4 previous similar messages [ 4223.457514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.464195] Lustre: Skipped 2 previous similar messages [ 4226.015277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4226.188466] Lustre: server umount lustre-MDT0000 complete [ 4228.583450] LustreError: 152560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.606391] LustreError: 152560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4228.855961] LustreError: 150256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873820 with bad export cookie 4057190602541447541 [ 4228.862282] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.866410] LustreError: 150256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4233.697923] LustreError: 150270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.698080] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.720028] Lustre: Skipped 1 previous similar message [ 4233.726164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4235.181206] Lustre: server umount lustre-MDT0001 complete [ 4252.129050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4252.223452] Lustre: server umount lustre-OST0000 complete [ 4258.260629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4265.255351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4292.583301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.772526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4292.794812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4292.864905] Lustre: lustre-MDT0000: new disk, initializing [ 4292.925166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.940614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4296.226563] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4304.539760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.623707] Lustre: 156517:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4304.699762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4304.702071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4304.711371] Lustre: Skipped 1 previous similar message [ 4304.774087] Lustre: lustre-MDT0001: new disk, initializing [ 4304.840811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4304.877341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4304.888364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4307.851937] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4313.477804] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.071823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4320.900321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4321.118351] Lustre: lustre-OST0000: new disk, initializing [ 4321.121565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4321.126222] Lustre: 158258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4321.186369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.497906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4328.501122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4328.513604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4328.597576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4331.385607] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4334.210899] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4334.350252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4337.313295] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4337.483347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4339.171835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.172695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.187379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.191670] Lustre: Skipped 1 previous similar message [ 4352.991176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4353.103684] Lustre: server umount lustre-OST0000 complete [ 4356.066636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4356.071664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.080071] Lustre: Skipped 1 previous similar message [ 4356.083901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.087322] Lustre: Skipped 4 previous similar messages [ 4370.399121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4370.490409] Lustre: server umount lustre-MDT0000 complete [ 4372.584637] LustreError: 158257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873964 with bad export cookie 4057190602541606371 [ 4372.585906] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.591112] LustreError: 158257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4372.758233] Lustre: server umount lustre-MDT0001 complete [ 4377.604556] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:26:09 (1779873969) [ 4387.393442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4392.822436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4413.277366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.408173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4413.425384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4413.466270] Lustre: lustre-MDT0000: new disk, initializing [ 4413.510717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.520320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4415.735514] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4421.495545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.557806] Lustre: 162827:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4421.694915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4424.023813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4428.688596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4430.770062] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4435.038208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.257017] Lustre: 164576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4438.938448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4443.453672] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4444.693144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4444.744627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4454.885691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.886081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.890197] LustreError: Skipped 1 previous similar message [ 4454.899493] Lustre: Skipped 1 previous similar message [ 4454.903133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.913488] Lustre: Skipped 7 previous similar messages [ 4458.674331] Lustre: server umount lustre-OST0000 complete [ 4462.562140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4465.122627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.128597] Lustre: Skipped 2 previous similar messages [ 4475.359122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.434106] Lustre: server umount lustre-MDT0000 complete [ 4477.098260] LustreError: 162821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874068 with bad export cookie 4057190602541607253 [ 4477.101239] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.103492] LustreError: 162821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4477.245776] Lustre: server umount lustre-MDT0001 complete [ 4481.908681] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:27:53 (1779874073) [ 4490.366947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4495.518894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4534.285240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.685914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4534.772845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4534.785235] Lustre: Skipped 3 previous similar messages [ 4534.991882] Lustre: lustre-MDT0000: new disk, initializing [ 4534.998853] Lustre: Skipped 2 previous similar messages [ 4535.196648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.210640] Lustre: Skipped 2 previous similar messages [ 4535.253401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4535.257827] Lustre: Skipped 2 previous similar messages [ 4544.352235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4556.021217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.093230] Lustre: 168749:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4556.475773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4560.654906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4568.920720] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4573.702335] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4582.223460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.520324] Lustre: 170503:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4587.917704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4590.699151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4590.843715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4594.973310] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4601.319216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.340304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.343187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4601.349698] Lustre: Skipped 7 previous similar messages [ 4607.445102] Lustre: server umount lustre-OST0000 complete [ 4616.813679] Lustre: server umount lustre-MDT0000 complete [ 4620.053502] LustreError: 168743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874211 with bad export cookie 4057190602541608331 [ 4620.054853] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4620.062616] LustreError: 168743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4620.333155] Lustre: server umount lustre-MDT0001 complete [ 4629.183333] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4631.477391] Key type lgssc unregistered [ 4631.809698] LNet: 172230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4631.815562] LNetError: 172230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4631.828795] LNet: Removed LNI 192.168.203.155@tcp [ 4632.603750] Key type .llcrypt unregistered [ 4632.607664] Key type ._llcrypt unregistered [ 4654.498430] Key type ._llcrypt registered [ 4654.500684] Key type .llcrypt registered [ 4654.615538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4655.409233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.458326] alg: No test for adler32 (adler32-zlib) [ 4656.529116] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4656.788553] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4658.463165] Key type lgssc registered [ 4659.368757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.812226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.830722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.279221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.239359] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4676.581677] LustreError: 173611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.611697] LustreError: 173611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4679.059173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.324913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4682.517167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4687.301314] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4691.210522] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4697.230409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.602305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.073506] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4708.067729] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4711.925330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4732.384661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.393284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4732.407624] Lustre: Skipped 1 previous similar message [ 4732.432961] Lustre: Skipped 1 previous similar message [ 4736.268354] Lustre: server umount lustre-OST0000 complete [ 4741.088868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4741.095809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.110842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.640435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.642091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.657993] Lustre: Skipped 1 previous similar message [ 4747.744841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4747.756338] Lustre: Skipped 1 previous similar message [ 4752.864971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4752.874301] Lustre: Skipped 1 previous similar message [ 4754.399475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4754.595294] Lustre: server umount lustre-MDT0000 complete [ 4757.732798] LustreError: 173592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874349 with bad export cookie 7520275620522313478 [ 4757.736390] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.743442] LustreError: 173592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4758.033137] Lustre: server umount lustre-MDT0001 complete [ 4767.171597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4769.345771] Key type lgssc unregistered [ 4769.554697] LNet: 176811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4769.561818] LNetError: 176811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4769.576940] LNet: Removed LNI 192.168.203.155@tcp [ 4770.258563] Key type .llcrypt unregistered [ 4770.260889] Key type ._llcrypt unregistered [ 4790.667146] Key type ._llcrypt registered [ 4790.669211] Key type .llcrypt registered [ 4790.796756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4791.475325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4791.538078] alg: No test for adler32 (adler32-zlib) [ 4792.540519] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4792.749776] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4794.423162] Key type lgssc registered [ 4795.447498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4802.206967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4802.229029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.603777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.319934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4813.797943] LustreError: 178203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.811562] LustreError: 178203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4816.059408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.559093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4821.137252] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4827.750772] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4831.273926] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4838.516832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4838.823962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.983460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4849.071495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4850.804634] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4858.971086] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 4859.374376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4859.377728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.752795] Lustre: server umount lustre-OST0000 complete [ 4869.305757] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93504a792800.ost_server_uuid 50 [ 4870.887894] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4884.838539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.841952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.850539] Lustre: Skipped 2 previous similar messages [ 4884.855988] Lustre: Skipped 2 previous similar messages [ 4890.081858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.091805] Lustre: Skipped 2 previous similar messages [ 4890.437766] Lustre: server umount lustre-MDT0000 complete [ 4893.576371] LustreError: 178183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874485 with bad export cookie 12082182733898362235 [ 4893.589419] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.592398] LustreError: 178183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4893.885112] Lustre: server umount lustre-MDT0001 complete [ 4901.381719] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:34:52 (1779874492) [ 4908.299657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.888044] LustreError: 181679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.025629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.150328] LustreError: 181680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4914.221983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4919.268403] LustreError: 181679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.184243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.532988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4926.576745] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4933.105630] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4937.735433] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4945.287252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4945.646401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.195488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4953.840973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4956.139683] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4961.912586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.072786] Lustre: lustre-OST0001: new disk, initializing [ 4962.076731] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4962.084885] Lustre: 184182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4962.154968] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4967.826652] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4971.093382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4971.103577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4971.166823] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4974.437672] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4977.634319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4977.648395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.663886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4979.680108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4979.694331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.736601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.741061] Lustre: Skipped 3 previous similar messages [ 4983.130632] Lustre: server umount lustre-OST0000 complete [ 4985.418262] LustreError: 183277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.771548] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4989.047604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4993.232806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4993.421126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4995.641056] LustreError: 183274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.658880] LustreError: 183274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5007.332573] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5007.339979] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.353203] LustreError: Skipped 1 previous similar message [ 5007.354548] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5007.403765] Lustre: Skipped 1 previous similar message [ 5012.463963] LustreError: 183672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.476984] LustreError: 183672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5012.832631] Lustre: server umount lustre-OST0001 complete [ 5017.574440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.602505] Lustre: Skipped 1 previous similar message [ 5027.808983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5027.823048] Lustre: Skipped 8 previous similar messages [ 5031.391183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.632292] Lustre: server umount lustre-MDT0000 complete [ 5035.951276] LustreError: 181661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874627 with bad export cookie 12082182733898363306 [ 5035.961185] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.968781] LustreError: 181661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5036.389392] Lustre: server umount lustre-MDT0001 complete [ 5059.012245] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:37:29 (1779874649) [ 5065.643275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.867966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5065.901258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.051210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.868911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5077.886841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.952871] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5077.977318] Lustre: 187471:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5082.115843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5087.991944] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5091.349776] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5097.415793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.575749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5103.567168] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5104.748748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5109.315756] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5116.060433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5116.140569] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5121.425557] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5123.172157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5126.260551] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5128.679315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.682482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5128.695362] Lustre: Skipped 1 previous similar message [ 5128.711295] Lustre: Skipped 1 previous similar message [ 5134.712431] Lustre: server umount lustre-OST0000 complete [ 5136.962000] LustreError: 189528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.979048] LustreError: 189528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5153.760711] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5156.832200] Lustre: server umount lustre-OST0001 complete [ 5164.516953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.532908] Lustre: Skipped 3 previous similar messages [ 5167.363129] Lustre: server umount lustre-MDT0000 complete [ 5171.276144] LustreError: 189115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874762 with bad export cookie 12082182733898364706 [ 5171.279675] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5171.284282] LustreError: 189115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5171.648464] Lustre: server umount lustre-MDT0001 complete [ 5194.357562] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:39:45 (1779874785) [ 5208.334384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5217.635229] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5249.582826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.779697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5249.804928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5249.881898] Lustre: lustre-MDT0000: new disk, initializing [ 5249.971315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.976911] Lustre: Skipped 3 previous similar messages [ 5250.007795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5253.481063] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5262.052520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.156345] Lustre: 195079:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5262.209107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5262.214511] Lustre: Skipped 1 previous similar message [ 5262.287451] Lustre: lustre-MDT0001: new disk, initializing [ 5262.395105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5262.409786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5266.264521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5273.801494] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5277.700594] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5283.832458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.124986] Lustre: lustre-OST0000: new disk, initializing [ 5284.131353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5284.139829] Lustre: 196819:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5289.859887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5293.089160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5293.102094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5293.184049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5296.542078] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5300.252755] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5300.481704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5304.549759] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5304.841525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5308.392238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.401699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5308.403694] Lustre: Skipped 1 previous similar message [ 5308.422198] Lustre: Skipped 9 previous similar messages [ 5312.808370] Lustre: server umount lustre-OST0000 complete [ 5318.624749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5318.636601] LustreError: Skipped 1 previous similar message [ 5322.908404] Lustre: server umount lustre-MDT0000 complete [ 5323.755726] LustreError: 195084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.775092] LustreError: 195084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5326.662777] LustreError: 195071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874918 with bad export cookie 12082182733898366498 [ 5326.665955] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.681081] LustreError: 195071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5327.029826] Lustre: server umount lustre-MDT0001 complete [ 5335.408980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.481573] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5347.461768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.714219] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5356.503136] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5360.435730] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5367.081375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.095722] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5377.964935] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5382.108927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5382.372792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5386.604919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5386.784740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5391.338384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.839259] Lustre: server umount lustre-OST0000 complete [ 5401.103454] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5405.189257] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5433.912788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.239634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.246058] Lustre: Skipped 5 previous similar messages [ 5439.337475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5445.368959] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5452.606451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5455.721329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.731906] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.732104] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5455.765024] Lustre: Skipped 1 previous similar message [ 5460.965379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.017965] Lustre: Skipped 4 previous similar messages [ 5461.048254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5461.050672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5461.078651] Lustre: Skipped 5 previous similar messages [ 5464.300900] Lustre: server umount lustre-OST0000 complete [ 5474.559062] Lustre: server umount lustre-MDT0000 complete [ 5476.320434] LustreError: 199600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.341884] LustreError: 199600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5478.402745] LustreError: 198870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875069 with bad export cookie 12082182733898367380 [ 5478.406986] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.415729] LustreError: 198870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5478.947754] Lustre: server umount lustre-MDT0001 complete [ 5488.111666] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:44:38 (1779875078) [ 5494.918277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.103779] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5508.962558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.320119] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5519.809166] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5524.215151] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5530.172375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5536.090778] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5542.867335] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.193415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5548.385140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5552.771801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5552.955700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5560.412349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.711357] Lustre: lustre-OST0001: new disk, initializing [ 5560.724277] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5560.749065] Lustre: 207332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5568.068180] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5569.227047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5569.257681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5569.518339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5578.166457] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5584.492205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5584.715648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5588.487639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5588.678658] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.106100] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.131710] LustreError: Skipped 1 previous similar message [ 5596.830786] Lustre: server umount lustre-OST0001 complete [ 5602.484063] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5602.605820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5606.268375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5606.414539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5633.821027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.149785] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5639.172035] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5639.185084] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5639.206616] Lustre: Skipped 1 previous similar message [ 5640.265771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5647.932620] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5649.932176] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5658.912217] Lustre: server umount lustre-OST0001 complete [ 5692.383128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5692.534622] Lustre: server umount lustre-OST0000 complete [ 5710.304564] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5710.500727] Lustre: server umount lustre-MDT0000 complete [ 5713.979689] LustreError: 204315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875305 with bad export cookie 12082182733898368339 [ 5713.990391] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.995535] LustreError: 204315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5720.410389] Lustre: server umount lustre-MDT0001 complete [ 5742.663447] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:48:53 (1779875333) [ 5749.515383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.827454] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5749.862352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5749.996276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.003057] Lustre: Skipped 5 previous similar messages [ 5753.731902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5762.567258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.785804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5762.867382] Lustre: 212285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5767.750542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5773.232830] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5777.451435] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5785.147177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5785.417269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5792.621572] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5798.325666] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5807.370212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5807.439604] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5813.761840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5819.685658] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5821.405932] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5825.776138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5825.967050] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5829.910903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5830.132313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5834.176811] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff935046bf7800.ost_server_uuid 50 [ 5835.744214] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff935046bf7800.ost_server_uuid in FULL state after 0 sec [ 5836.647722] Lustre: Permanently deactivating lustre-OST0001 [ 5836.660676] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5839.618885] Lustre: Permanently reactivating lustre-OST0001 [ 5846.505714] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5846.506150] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.512172] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5846.512180] Lustre: Skipped 21 previous similar messages [ 5846.526769] LustreError: Skipped 5 previous similar messages [ 5846.578556] Lustre: Skipped 12 previous similar messages [ 5849.512840] Lustre: server umount lustre-OST0001 complete [ 5851.621716] LustreError: 213907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.641574] LustreError: 213907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 5859.543734] Lustre: server umount lustre-OST0000 complete [ 5869.702279] Lustre: server umount lustre-MDT0000 complete [ 5874.206903] LustreError: 213019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875465 with bad export cookie 12082182733898369634 [ 5874.213557] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.225412] LustreError: 213019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5874.695594] Lustre: server umount lustre-MDT0001 complete [ 5895.958415] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:51:26 (1779875486) [ 5903.417508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.706503] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5903.746881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5903.764505] Lustre: Skipped 5 previous similar messages [ 5908.805628] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5917.174635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.268581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5917.297749] Lustre: 218396:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5921.476363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5926.252878] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5929.717405] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5935.459207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5935.626589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5935.646903] Lustre: Permanently deactivating lustre-OST0000 [ 5935.659621] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5941.300276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5941.511868] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5946.829864] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5953.310309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.375555] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5959.275858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5963.396227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5964.463771] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5965.946058] Lustre: Permanently reactivating lustre-OST0000 [ 5965.957923] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5965.962591] Lustre: Skipped 2 previous similar messages [ 5968.919699] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5968.928244] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.935924] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5968.963788] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5968.982688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5984.542644] Lustre: server umount lustre-OST0001 complete [ 5993.670530] Lustre: server umount lustre-OST0000 complete [ 6004.131547] Lustre: server umount lustre-MDT0000 complete [ 6008.030876] LustreError: 218386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875599 with bad export cookie 12082182733898371286 [ 6008.031912] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.052633] LustreError: 218386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6008.474458] Lustre: server umount lustre-MDT0001 complete [ 6019.782645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6022.371455] Key type lgssc unregistered [ 6022.662986] LNet: 222805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.669135] LNetError: 222805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.682466] LNet: Removed LNI 192.168.203.155@tcp [ 6023.642320] Key type .llcrypt unregistered [ 6023.644892] Key type ._llcrypt unregistered [ 6045.565962] Key type ._llcrypt registered [ 6045.573473] Key type .llcrypt registered [ 6045.659429] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6046.603526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6046.769721] alg: No test for adler32 (adler32-zlib) [ 6047.838753] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6048.055224] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6049.743381] Key type lgssc registered [ 6050.659669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6058.910130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6058.932289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.321713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.802879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6069.742537] LustreError: 224255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.773937] LustreError: 224255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6071.034322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.157721] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6071.188749] Lustre: 224243:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6071.201169] Lustre: Permanently deactivating lustre-MDT0001 [ 6071.215455] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6071.558975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.493875] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6074.498672] LustreError: 224285:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6074.510918] LustreError: 224285:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6074.523865] LustreError: 224285:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6075.405990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6080.949154] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6084.686753] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6091.525795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.920131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.034587] LustreError: 225869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.067774] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6098.062311] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6101.485374] LustreError: 225869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.994596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6104.547847] LustreError: 225869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.568912] LustreError: 225869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6104.577503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6109.668727] LustreError: 225870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.679163] LustreError: 225870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6111.188648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6111.432479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6116.869772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6116.885869] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6117.304321] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6122.671643] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6126.477116] Lustre: Permanently reactivating lustre-MDT0001 [ 6126.497913] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6126.502652] Lustre: Skipped 2 previous similar messages [ 6131.795847] Lustre: lustre-MDT0001: Client 2fd3d85e-6a1f-460c-976b-84cdc896dc36 (at 192.168.203.55@tcp) reconnecting [ 6135.605426] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6137.309791] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6142.661685] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6142.930462] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6147.366442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6147.529981] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6150.233749] Lustre: Permanently deactivating lustre-MDT0001 [ 6150.245289] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6150.249968] Lustre: Skipped 2 previous similar messages [ 6159.479119] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6167.218385] Lustre: server umount lustre-MDT0000 complete [ 6168.035364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.072375] LustreError: 224250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.073533] Lustre: Skipped 2 previous similar messages [ 6168.115389] LustreError: 224250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6170.788072] LustreError: 224234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875762 with bad export cookie 5363265855844021211 [ 6170.798634] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.807064] LustreError: 224234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.240565] Lustre: server umount lustre-MDT0001 complete [ 6175.140484] Lustre: server umount lustre-OST0000 complete [ 6187.321497] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:56:17 (1779875777) [ 6197.719538] Lustre: server umount lustre-OST0001 complete [ 6204.211660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6213.398814] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6247.829213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.119372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6248.126293] Lustre: Skipped 2 previous similar messages [ 6248.156780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6248.229086] Lustre: lustre-MDT0000: new disk, initializing [ 6248.330102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.357940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6251.837574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6261.020990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.106415] Lustre: 231910:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6261.180821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6261.184090] Lustre: Skipped 1 previous similar message [ 6261.261765] Lustre: lustre-MDT0001: new disk, initializing [ 6261.345151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.369128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6261.376602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6265.332931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6272.592857] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6276.167563] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6282.692943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.937708] Lustre: lustre-OST0000: new disk, initializing [ 6282.942334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6282.950894] Lustre: 233651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6283.041662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.537411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6292.529482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6292.534813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6292.606595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6295.163896] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6297.045557] LustreError: 233658:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6304.428425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6304.590385] Lustre: lustre-OST0001: new disk, initializing [ 6304.594108] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6304.601590] Lustre: 234654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6307.111841] LustreError: 233658:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6307.142266] LustreError: 234556:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6309.619496] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6309.928081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6309.935882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6310.012911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6315.532437] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6316.890853] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6317.199881] LustreError: 234556:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6320.098070] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.103909] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.106873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6320.123754] Lustre: Skipped 2 previous similar messages [ 6320.135347] Lustre: Skipped 1 previous similar message [ 6321.756766] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 6325.231029] LustreError: 233635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.236441] Lustre: server umount lustre-OST0001 complete [ 6325.250852] LustreError: 233635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6343.652856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6343.657269] LustreError: Skipped 1 previous similar message [ 6343.659489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.672460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6345.698170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6345.707204] Lustre: Skipped 2 previous similar messages [ 6347.719106] Lustre: server umount lustre-OST0000 complete [ 6353.375840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6353.389119] LustreError: Skipped 1 previous similar message [ 6353.395332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.405895] Lustre: Skipped 1 previous similar message [ 6353.413340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.529230] Lustre: server umount lustre-MDT0000 complete [ 6361.056383] LustreError: 231924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.097650] LustreError: 231924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6362.144151] LustreError: 234653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875953 with bad export cookie 5363265855844024025 [ 6362.154970] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6362.162061] LustreError: 234653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6368.847876] Lustre: server umount lustre-MDT0001 complete [ 6379.578398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6382.326325] Key type lgssc unregistered [ 6382.641640] LNet: 236555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6382.653505] LNetError: 236555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6382.686205] LNet: Removed LNI 192.168.203.155@tcp [ 6383.234851] Key type .llcrypt unregistered [ 6383.239182] Key type ._llcrypt unregistered [ 6420.559951] Key type ._llcrypt registered [ 6420.566945] Key type .llcrypt registered [ 6420.737296] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6422.188707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6422.263335] alg: No test for adler32 (adler32-zlib) [ 6423.416326] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6423.772824] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6425.415353] Key type lgssc registered [ 6426.536610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6434.500636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6434.520497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.815530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6435.851308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6435.958587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.247484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6447.926733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.028736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6448.077322] Lustre: 238744:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6448.328965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6452.210158] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6457.924275] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6462.231793] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6468.902182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6469.175721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6469.472405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6475.223794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6479.518163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6481.037866] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6535.650940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.655210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.683165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.162205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.190598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.203654] Lustre: Skipped 1 previous similar message [ 6538.727431] Lustre: server umount lustre-OST0000 complete [ 6543.295552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6555.542308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.923900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.830428] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6561.085497] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.108956] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6566.177076] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6576.486890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.502748] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6607.339576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6607.343022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.345075] LustreError: Skipped 1 previous similar message [ 6607.349309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.381426] Lustre: Skipped 1 previous similar message [ 6610.643965] Lustre: server umount lustre-OST0000 complete [ 6617.570199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.573955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6617.592262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.620880] Lustre: Skipped 2 previous similar messages [ 6622.693494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6622.698875] Lustre: Skipped 1 previous similar message [ 6628.325981] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6628.451539] Lustre: server umount lustre-MDT0000 complete [ 6632.444479] LustreError: 238738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876223 with bad export cookie 18553909133391205 [ 6632.452250] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6632.454412] LustreError: 238738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6632.862572] Lustre: server umount lustre-MDT0001 complete [ 6642.088505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6644.072056] Key type lgssc unregistered [ 6644.297307] LNet: 243708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6644.308732] LNetError: 243708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6644.319407] LNet: Removed LNI 192.168.203.155@tcp [ 6644.948141] Key type .llcrypt unregistered [ 6644.949561] Key type ._llcrypt unregistered [ 6664.770384] Key type ._llcrypt registered [ 6664.772157] Key type .llcrypt registered [ 6664.849692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6665.627514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6665.735864] alg: No test for adler32 (adler32-zlib) [ 6666.790933] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6667.019584] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6668.671172] Key type lgssc registered [ 6669.457484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6675.522809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6675.541047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6676.988701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.702976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6687.213473] LustreError: 245087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.252078] LustreError: 245087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6690.687314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.039858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.378299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6701.814887] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6705.829565] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6712.537976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.047282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.081432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6722.238822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6725.655197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6752.224101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.229849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.243061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6753.265513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.265630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6753.266186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6754.710524] Lustre: server umount lustre-OST0000 complete [ 6758.379439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.397444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6758.407223] Lustre: Skipped 2 previous similar messages [ 6763.490593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6764.243382] Lustre: server umount lustre-MDT0000 complete [ 6766.792195] LustreError: 245067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876358 with bad export cookie 15222102430082253259 [ 6766.800383] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6766.801135] LustreError: 245067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6767.001075] Lustre: server umount lustre-MDT0001 complete [ 6775.525360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6778.067648] Key type lgssc unregistered [ 6778.333835] LNet: 249036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6778.340831] LNetError: 249036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6778.357375] LNet: Removed LNI 192.168.203.155@tcp [ 6779.041590] Key type .llcrypt unregistered [ 6779.046223] Key type ._llcrypt unregistered [ 6793.965313] Key type ._llcrypt registered [ 6793.967646] Key type .llcrypt registered [ 6794.070798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6794.803390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6794.865669] alg: No test for adler32 (adler32-zlib) [ 6795.923486] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6796.133452] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6797.807162] Key type lgssc registered [ 6798.800802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6805.073307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6805.091410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.561825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.368410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6816.739153] LustreError: 250235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.756050] LustreError: 250235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6817.585717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.877401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6821.355304] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6826.581310] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6830.491088] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6836.847673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6837.135379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.437175] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6847.969884] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6851.521259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6863.845395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.863901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.881072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.915794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.917509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.930396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.260824] Lustre: server umount lustre-OST0000 complete [ 6872.035572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.055063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.065244] Lustre: Skipped 2 previous similar messages [ 6877.163850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6877.171029] Lustre: Skipped 1 previous similar message [ 6877.343445] Lustre: server umount lustre-MDT0000 complete [ 6880.497892] LustreError: 250216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876472 with bad export cookie 16365205227730832292 [ 6880.504498] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6880.515089] LustreError: 250216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6880.779268] Lustre: server umount lustre-MDT0001 complete [ 6889.722866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6892.123187] Key type lgssc unregistered [ 6892.400512] LNet: 253555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6892.411602] LNetError: 253555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6892.428246] LNet: Removed LNI 192.168.203.155@tcp [ 6893.139335] Key type .llcrypt unregistered [ 6893.144148] Key type ._llcrypt unregistered [ 6913.753201] Key type ._llcrypt registered [ 6913.756989] Key type .llcrypt registered [ 6913.870985] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6914.842243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6914.859668] alg: No test for adler32 (adler32-zlib) [ 6915.912669] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6916.086102] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6917.767217] Key type lgssc registered [ 6919.303823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6928.363111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6928.397186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.114690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.869573] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6939.617115] LustreError: 254956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.634863] LustreError: 254956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6941.611036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.043405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6946.550260] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6951.884316] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.252716] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6964.633300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.891325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.637891] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6976.183563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6977.152506] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6997.985551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.992834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.005450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.094818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.095242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.110707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.916443] Lustre: server umount lustre-OST0000 complete [ 7008.738608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7008.748487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.773994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7008.786347] Lustre: Skipped 1 previous similar message [ 7012.325738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.326628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.859981] Lustre: server umount lustre-MDT0000 complete [ 7017.218650] LustreError: 254938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876608 with bad export cookie 1566412804431248454 [ 7017.219802] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.239774] LustreError: 254938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7017.595516] Lustre: server umount lustre-MDT0001 complete [ 7029.255074] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7032.602925] Key type lgssc unregistered [ 7032.889582] LNet: 258905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7032.925129] LNetError: 258905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7033.967670] LNet: Removed LNI 192.168.203.155@tcp [ 7035.004968] Key type .llcrypt unregistered [ 7035.007347] Key type ._llcrypt unregistered [ 7056.286587] Key type ._llcrypt registered [ 7056.290316] Key type .llcrypt registered [ 7056.400488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7057.306969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7057.372841] alg: No test for adler32 (adler32-zlib) [ 7058.413244] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 7058.711049] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7060.408254] Key type lgssc registered [ 7061.509664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7067.782437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7067.799454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.375759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.111261] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7074.784538] LustreError: 260124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.808377] LustreError: 260124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7078.888650] LustreError: 260126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7083.488887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.884835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.470183] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7093.846499] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7097.861268] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7104.140048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7104.503304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7110.248494] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7113.712683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7115.263912] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.321666] Lustre: server umount lustre-OST0000 complete [ 7134.178068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.179791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.185744] Lustre: Skipped 1 previous similar message [ 7134.191906] Lustre: Skipped 1 previous similar message [ 7139.296436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.304575] Lustre: Skipped 1 previous similar message [ 7142.437069] Lustre: server umount lustre-MDT0000 complete [ 7144.417887] LustreError: 260109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.449123] LustreError: 260109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7145.440929] Lustre: 259423:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779876720/real 1779876720] req@000000002c829892 x1866335938175744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779876736 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7145.480299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.161489] LustreError: 260093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876737 with bad export cookie 4863724056384587227 [ 7146.165568] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.174333] LustreError: 260093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7146.554323] Lustre: server umount lustre-MDT0001 complete [ 7156.634895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7159.276760] Key type lgssc unregistered [ 7159.549874] LNet: 263468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7159.567343] LNetError: 263468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7159.587834] LNet: Removed LNI 192.168.203.155@tcp [ 7160.389839] Key type .llcrypt unregistered [ 7160.394280] Key type ._llcrypt unregistered [ 7171.844775] Key type ._llcrypt registered [ 7171.851861] Key type .llcrypt registered [ 7171.953159] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7187.467636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7201.136388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7202.158576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7202.171488] alg: No test for adler32 (adler32-zlib) [ 7203.193975] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 7203.416994] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7205.119155] Key type lgssc registered [ 7206.295930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7239.390624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7239.417990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.666181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7240.698354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7240.760197] Lustre: lustre-MDT0000: new disk, initializing [ 7240.872185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.892174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7244.997784] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7256.375748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.540745] Lustre: 266969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.592521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7256.596872] Lustre: Skipped 1 previous similar message [ 7256.665775] Lustre: lustre-MDT0001: new disk, initializing [ 7256.784596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.820350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7256.838421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7261.824310] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7270.306350] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7274.258661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7281.568704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7281.819364] Lustre: lustre-OST0000: new disk, initializing [ 7281.822461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7281.828361] Lustre: 268711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7281.900593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.368643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7289.898570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7289.905709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7290.016514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7296.715514] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7301.133947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7301.378345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7305.957517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7306.088191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7310.311353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7310.320467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.332315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.075161] Lustre: server umount lustre-OST0000 complete [ 7318.498730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7318.503840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.522757] Lustre: Skipped 1 previous similar message [ 7318.537388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7318.544883] Lustre: Skipped 1 previous similar message [ 7321.059150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7321.060517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.492919] Lustre: server umount lustre-MDT0000 complete [ 7326.805259] LustreError: 266959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876918 with bad export cookie 3943106643455674625 [ 7326.805620] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7326.819175] LustreError: 266959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7327.256106] Lustre: server umount lustre-MDT0001 complete [ 7334.704125] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:15:25 (1779876925) [ 7340.064603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.412414] LustreError: 270973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.426605] LustreError: 270973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7340.479924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7344.673813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7345.634211] LustreError: 270974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.731688] LustreError: 270973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.202613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.628049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.712799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7362.923208] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7366.305896] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7373.460863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7373.778456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.884545] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7389.085987] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7400.421705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.425996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7400.452895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7400.452903] Lustre: Skipped 1 previous similar message [ 7405.536914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.555275] Lustre: Skipped 2 previous similar messages [ 7405.745707] Lustre: server umount lustre-OST0000 complete [ 7410.148334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7410.160213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.168371] Lustre: Skipped 1 previous similar message [ 7414.940796] Lustre: server umount lustre-MDT0000 complete [ 7415.777868] LustreError: 272181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.793105] LustreError: 272181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7418.057634] LustreError: 271691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877009 with bad export cookie 3943106643455675507 [ 7418.066471] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7418.076919] LustreError: 271691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7418.320268] Lustre: server umount lustre-MDT0001 complete [ 7427.764316] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7430.169963] Key type lgssc unregistered [ 7430.406794] LNet: 274151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7430.413261] LNetError: 274151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7430.434343] LNet: Removed LNI 192.168.203.155@tcp [ 7431.187187] Key type .llcrypt unregistered [ 7431.188634] Key type ._llcrypt unregistered [ 7457.315670] Key type ._llcrypt registered [ 7457.317963] Key type .llcrypt registered [ 7457.397551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7472.273343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7473.222912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7473.324620] alg: No test for adler32 (adler32-zlib) [ 7474.399886] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 7474.600203] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7476.239137] Key type lgssc registered [ 7477.174295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7520.040578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7520.095801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.523778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7521.571422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7521.702637] Lustre: lustre-MDT0000: new disk, initializing [ 7521.809800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.842304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7525.904594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7535.719722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.829504] Lustre: 277944:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7535.873940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7535.877623] Lustre: Skipped 1 previous similar message [ 7535.929944] Lustre: lustre-MDT0001: new disk, initializing [ 7536.006850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7536.061466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7536.078967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7539.811137] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7547.905730] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7552.467723] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7560.942066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7561.250849] Lustre: lustre-OST2710: new disk, initializing [ 7561.254906] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7561.266482] Lustre: 279688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7561.335210] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7568.406305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7568.413722] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7568.508834] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7568.675902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7578.585350] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7586.516939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.734571] Lustre: lustre-OST03e8: new disk, initializing [ 7586.740470] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7586.749584] Lustre: 280644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7586.868831] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7593.335665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7594.625392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7594.635155] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7594.694167] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7600.251639] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7608.337987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7608.548444] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7613.459365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7613.739945] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7619.177199] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7619.543836] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7624.354519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7624.634729] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7630.822685] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.830408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.843520] Lustre: Skipped 3 previous similar messages [ 7630.858768] Lustre: Skipped 3 previous similar messages [ 7631.510232] Lustre: server umount lustre-MDT0000 complete [ 7635.258239] LustreError: 280643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877226 with bad export cookie 2557095497552785315 [ 7635.260685] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.280791] LustreError: 280643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7635.794908] Lustre: server umount lustre-MDT0001 complete [ 7650.231574] Lustre: server umount lustre-OST2710 complete [ 7652.192689] Lustre: 275742:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779877227/real 1779877227] req@0000000031d78650 x1866336374425984/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779877243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7652.224648] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.970502] Lustre: server umount lustre-OST03e8 complete [ 7674.320379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7684.280717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7721.291961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.501974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7721.525126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7721.584670] Lustre: lustre-MDT0000: new disk, initializing [ 7721.642962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7721.670063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7725.565099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7735.830934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.947334] Lustre: 285677:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7736.003245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7736.008746] Lustre: Skipped 1 previous similar message [ 7736.093877] Lustre: lustre-MDT0001: new disk, initializing [ 7736.218808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7736.258763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7736.274391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7740.792303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7748.265320] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7752.047696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7760.077165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.448131] Lustre: lustre-OST0000: new disk, initializing [ 7760.451456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7760.458275] Lustre: 287418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7760.575861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.329360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7767.684311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7767.699418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7767.814486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7774.315531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7778.423866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7778.661733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7782.685030] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7782.920281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7788.514786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7788.522960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.526956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7788.534504] Lustre: Skipped 1 previous similar message [ 7793.638324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7793.651297] Lustre: Skipped 2 previous similar messages [ 7798.754209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.760312] Lustre: Skipped 1 previous similar message [ 7799.263141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7799.405474] Lustre: server umount lustre-OST0000 complete [ 7802.852399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7802.865651] LustreError: Skipped 1 previous similar message [ 7802.869716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.889483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.949480] Lustre: server umount lustre-MDT0000 complete [ 7812.463372] LustreError: 285669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877403 with bad export cookie 2557095497552786995 [ 7812.482571] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.486417] LustreError: 285669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7813.034686] Lustre: server umount lustre-MDT0001 complete [ 7820.444856] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:23:31 (1779877411) [ 7821.801393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7823.298752] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:23:34 (1779877414) [ 7825.981702] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7851.156491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7851.425617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7851.448435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7851.563980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.549316] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7865.692314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.878788] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7865.960403] Lustre: 291115:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7871.615221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7893.432828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.735021] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 7893.753686] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7893.762818] LustreError: 292534:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 7893.769386] LustreError: 292534:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7893.780945] LustreError: 292534:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7893.825976] Lustre: server umount lustre-OST0000 complete [ 7893.829304] LustreError: 292534:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7901.164235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.168355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.172794] Lustre: Skipped 2 previous similar messages [ 7901.190558] Lustre: Skipped 3 previous similar messages [ 7905.602582] Lustre: server umount lustre-MDT0000 complete [ 7906.274676] LustreError: 291120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.294110] LustreError: 291120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7908.863242] LustreError: 291105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877500 with bad export cookie 2557095497552787870 [ 7908.877595] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7908.879897] LustreError: 291105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7909.143362] Lustre: server umount lustre-MDT0001 complete [ 7919.262838] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7922.017862] Key type lgssc unregistered [ 7922.285665] LNet: 293638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7922.292945] LNetError: 293638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7923.368134] LNet: Removed LNI 192.168.203.155@tcp [ 7924.263473] Key type .llcrypt unregistered [ 7924.265720] Key type ._llcrypt unregistered [ 7935.130441] Key type ._llcrypt registered [ 7935.134748] Key type .llcrypt registered [ 7935.262520] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7935.934636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7935.975030] alg: No test for adler32 (adler32-zlib) [ 7937.108724] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 7937.388573] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7939.015305] Key type lgssc registered [ 7940.055031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7985.777158] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7994.032729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7994.062624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.350359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7995.373670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7995.572335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.207565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8006.840775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.927345] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8006.954280] Lustre: 296369:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8007.235418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8011.069767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8016.521775] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8020.409231] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8039.798188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.041842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8040.466379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.863533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8054.758533] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8061.408859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8061.424165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8061.431312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8061.436420] Lustre: Skipped 1 previous similar message [ 8061.451365] Lustre: Skipped 1 previous similar message [ 8063.221919] Lustre: server umount lustre-OST0000 complete [ 8066.528649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8066.545755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8066.548696] Lustre: Skipped 1 previous similar message [ 8071.648669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8071.655945] Lustre: Skipped 1 previous similar message [ 8072.573873] Lustre: server umount lustre-MDT0000 complete [ 8075.986443] LustreError: 296361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877667 with bad export cookie 9673223379810292494 [ 8075.989654] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8075.998098] LustreError: 296361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8076.260491] Lustre: server umount lustre-MDT0001 complete [ 8086.596980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8089.597339] Key type lgssc unregistered [ 8089.903933] LNet: 299583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8089.917515] LNetError: 299583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8089.945253] LNet: Removed LNI 192.168.203.155@tcp [ 8090.892695] Key type .llcrypt unregistered [ 8090.899298] Key type ._llcrypt unregistered [ 8113.649987] Key type ._llcrypt registered [ 8113.657153] Key type .llcrypt registered [ 8113.747348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8114.964125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8114.996280] alg: No test for adler32 (adler32-zlib) [ 8116.080278] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8116.380249] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8118.039297] Key type lgssc registered [ 8119.135692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8126.164233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8126.187159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.750544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.391438] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8138.211397] LustreError: 300984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.251914] LustreError: 300984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8139.474380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.870320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.285882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8148.393971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8151.931108] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8160.702611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.020797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.404368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8171.466536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8183.783568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.785833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.791320] Lustre: Skipped 2 previous similar messages [ 8183.800579] Lustre: Skipped 2 previous similar messages [ 8187.474645] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 8188.898188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.906021] Lustre: Skipped 1 previous similar message [ 8191.434520] Lustre: server umount lustre-MDT0000 complete [ 8192.572602] LustreError: 300980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.017030] LustreError: 300984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.032669] LustreError: 300984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8194.443207] LustreError: 300962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877785 with bad export cookie 248127593265991485 [ 8194.447586] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.452388] LustreError: 300962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8197.686182] LustreError: 300979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.719453] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 8199.137304] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.837738] Lustre: server umount lustre-MDT0001 complete [ 8207.344598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8216.638874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.901227] LustreError: 302589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.927434] LustreError: 302589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8220.011288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.084958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8224.337973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8229.159747] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8229.166927] LustreError: 302590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.195688] LustreError: 302590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8231.996550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.256924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.905076] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8238.567593] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8240.058534] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8241.321679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8244.439031] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8245.945879] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8250.851243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8250.861753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.873468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8250.880453] Lustre: Skipped 1 previous similar message [ 8252.898979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8252.905885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.427215] Lustre: server umount lustre-OST0000 complete [ 8263.142512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8263.154624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.164365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8263.168453] Lustre: Skipped 3 previous similar messages [ 8265.579899] Lustre: server umount lustre-MDT0000 complete [ 8268.827104] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877860 with bad export cookie 248127593265999108 [ 8268.831660] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8268.834877] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8269.125978] Lustre: server umount lustre-MDT0001 complete [ 8278.947934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8281.555444] Key type lgssc unregistered [ 8281.793247] LNet: 306683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8281.803402] LNetError: 306683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8281.828058] LNet: Removed LNI 192.168.203.155@tcp [ 8282.945275] Key type .llcrypt unregistered [ 8282.947962] Key type ._llcrypt unregistered [ 8305.539743] Key type ._llcrypt registered [ 8305.542070] Key type .llcrypt registered [ 8305.631273] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8306.268442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8306.311632] alg: No test for adler32 (adler32-zlib) [ 8307.262345] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8307.436703] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8309.071236] Key type lgssc registered [ 8310.129768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8316.740605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8316.755609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.063738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8320.873172] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8326.916141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.206721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8329.961473] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8334.843603] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8338.557195] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8342.496694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.501972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8342.502261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.507045] Lustre: Skipped 1 previous similar message [ 8347.621362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8347.631438] Lustre: Skipped 2 previous similar messages [ 8347.755459] Lustre: server umount lustre-MDT0000 complete [ 8351.038743] LustreError: 308043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877942 with bad export cookie 15577648659564962542 [ 8351.049542] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8351.054817] LustreError: 308043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8351.428179] Lustre: server umount lustre-MDT0001 complete [ 8363.121282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.340651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8363.372711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8363.607811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.048879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8374.374977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.467906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8374.517726] Lustre: 310424:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8374.756668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8378.433846] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8382.905177] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8386.365080] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8394.605358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.801429] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8394.813926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8394.827842] LustreError: 312081:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 8394.842976] LustreError: 312081:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8394.851644] LustreError: 312081:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8394.922038] Lustre: server umount lustre-OST0000 complete [ 8394.928587] LustreError: 312081:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8403.880691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.049966] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8404.192582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.368756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8413.058887] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8414.307168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8419.452764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.531992] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8419.639210] Lustre: lustre-OST0001: new disk, initializing [ 8419.642436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8419.647500] Lustre: 313506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8419.725323] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.963473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8422.971371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8423.063475] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8424.684688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8429.424926] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8433.632241] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.639854] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.659418] LustreError: Skipped 1 previous similar message [ 8433.668994] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8437.981970] Lustre: server umount lustre-OST0001 complete [ 8438.762689] LustreError: 313713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.776933] LustreError: 313713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8443.880233] LustreError: 312549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.889451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.890409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.890414] Lustre: Skipped 1 previous similar message [ 8443.893774] LustreError: 312549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8443.926651] Lustre: Skipped 2 previous similar messages [ 8447.253716] Lustre: server umount lustre-OST0000 complete [ 8451.554102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8451.557937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.566337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.568752] Lustre: Skipped 1 previous similar message [ 8456.378645] Lustre: server umount lustre-MDT0000 complete [ 8459.234289] LustreError: 310434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.255945] LustreError: 310434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8459.481266] LustreError: 311147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878050 with bad export cookie 15577648659564963102 [ 8459.485249] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.499076] LustreError: 311147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8459.798288] Lustre: server umount lustre-MDT0001 complete [ 8470.643082] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8473.466065] Key type lgssc unregistered [ 8473.865192] LNet: 315337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8473.874744] LNetError: 315337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8473.893784] LNet: Removed LNI 192.168.203.155@tcp [ 8474.529288] Key type .llcrypt unregistered [ 8474.531027] Key type ._llcrypt unregistered [ 8541.435249] Key type ._llcrypt registered [ 8541.438360] Key type .llcrypt registered [ 8541.528841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8554.613222] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8555.725168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8555.745143] alg: No test for adler32 (adler32-zlib) [ 8556.771646] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8556.985682] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8558.631172] Key type lgssc registered [ 8559.528314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8587.740663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8587.768929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.979194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8589.015225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8589.085312] Lustre: lustre-MDT0000: new disk, initializing [ 8589.182280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8589.216550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8593.103767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8602.232366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.330441] Lustre: 320238:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8602.404589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8602.411817] Lustre: Skipped 1 previous similar message [ 8602.534823] Lustre: lustre-MDT0001: new disk, initializing [ 8602.658528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.704356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.711610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8606.560197] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8613.200140] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8616.490608] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8622.640558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.837912] Lustre: lustre-OST0000: new disk, initializing [ 8622.841172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8622.845698] Lustre: 321977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.918216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8628.337715] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8629.292613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8629.301876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8629.372060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8634.502251] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8638.492907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8638.734352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8643.353540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8643.557096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8649.701122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.704465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.709584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.714894] Lustre: Skipped 1 previous similar message [ 8651.785248] Lustre: server umount lustre-OST0000 complete [ 8658.918178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8658.935527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.971936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8658.986456] Lustre: Skipped 1 previous similar message [ 8659.945123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.983834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.997524] Lustre: Skipped 1 previous similar message [ 8662.544409] Lustre: server umount lustre-MDT0000 complete [ 8667.570841] LustreError: 320229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878259 with bad export cookie 9610663506836595845 [ 8667.581504] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8667.601738] LustreError: 320229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8668.244480] Lustre: server umount lustre-MDT0001 complete [ 8678.011980] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:37:48 (1779878268) [ 8685.602872] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:37:56 (1779878276) [ 8693.245584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.706527] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8693.733899] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8693.793411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8698.197477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8698.849122] LustreError: 324585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.975434] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.521053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.847867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.389515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8713.900595] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8717.099867] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8723.468133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.827183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.551853] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8734.848837] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8747.090335] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8748.856284] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8750.420467] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8752.291910] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8753.920697] Lustre: Failing over lustre-MDT0000 [ 8754.302972] Lustre: server umount lustre-MDT0000 complete [ 8754.662577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.669790] Lustre: Skipped 1 previous similar message [ 8754.674041] LustreError: 324580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.687507] LustreError: 324580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8759.779641] LustreError: 324581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.814680] LustreError: 324581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8770.022794] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.048108] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8771.039340] Lustre: 318461:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779878346/real 1779878346] req@000000005a2fc32a x1866337508983936/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779878362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8771.079173] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8772.085809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.291694] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x855fef8a3cce2c11 [ 8781.310711] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 8781.557171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.606631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8782.389614] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8785.783682] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8786.933497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8786.963265] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8787.018788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8792.236282] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8793.590375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8796.829959] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8798.501908] Lustre: Failing over lustre-MDT0000 [ 8798.770269] Lustre: server umount lustre-MDT0000 complete [ 8802.275578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.280077] LustreError: 324579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8802.286725] Lustre: Skipped 2 previous similar messages [ 8802.302330] LustreError: 324579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8802.304545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8817.073251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.231204] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8817.426192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.503605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8818.230132] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8821.538903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8822.769334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8822.784238] Lustre: Skipped 2 previous similar messages [ 8822.809551] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8822.862902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8829.985566] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8831.681669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8838.112100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8838.126990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.133580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.148044] Lustre: Skipped 1 previous similar message [ 8843.128950] Lustre: server umount lustre-MDT0000 complete [ 8843.232493] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.253367] LustreError: 324584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8846.172975] LustreError: 324564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878437 with bad export cookie 9610663506836598568 [ 8846.184565] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.200783] LustreError: 324564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8846.444842] Lustre: server umount lustre-MDT0001 complete [ 8859.861886] Lustre: server umount lustre-OST0000 complete [ 8878.138543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.650589] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8888.923228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.220464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.227683] Lustre: Skipped 1 previous similar message [ 8893.333959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8898.260391] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8901.594640] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8907.754587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.626424] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8917.272340] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8923.385402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8933.859814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.872143] Lustre: Skipped 1 previous similar message [ 8933.876424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8933.876790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8933.887858] Lustre: Skipped 3 previous similar messages [ 8939.403293] Lustre: server umount lustre-OST0000 complete [ 8944.124688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8944.131929] Lustre: Skipped 3 previous similar messages [ 8951.710900] Lustre: server umount lustre-MDT0000 complete [ 8954.338111] LustreError: 330375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.363629] LustreError: 330375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8955.317025] LustreError: 330355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878546 with bad export cookie 9610663506836599317 [ 8955.324173] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8955.329837] LustreError: 330355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8955.535947] Lustre: server umount lustre-MDT0001 complete [ 8967.404229] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8970.304341] Key type lgssc unregistered [ 8970.505774] LNet: 333571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8970.529532] LNetError: 333571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8970.545986] LNet: Removed LNI 192.168.203.155@tcp [ 8971.230844] Key type .llcrypt unregistered [ 8971.232614] Key type ._llcrypt unregistered [ 8995.850124] Key type ._llcrypt registered [ 8995.851848] Key type .llcrypt registered [ 8995.938192] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8996.949881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8996.994532] alg: No test for adler32 (adler32-zlib) [ 8998.077510] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8998.321791] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9000.007147] Key type lgssc registered [ 9000.906792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9008.031418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9008.047658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.414948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.131392] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9018.858796] LustreError: 335070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.877882] LustreError: 335070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9019.744608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9020.078636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9024.096864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9028.823602] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9032.097339] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9037.723549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9038.155337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9044.219667] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9049.725716] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9053.552955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9066.469741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9066.489301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9066.515368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.024840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.030686] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.315469] Lustre: server umount lustre-MDT0000 complete [ 9074.145909] LustreError: 335781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.164033] LustreError: 335781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9074.906768] LustreError: 335050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878666 with bad export cookie 17020885309785123498 [ 9074.916322] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9074.919246] LustreError: 335050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9075.234610] Lustre: server umount lustre-MDT0001 complete [ 9089.241889] Lustre: server umount lustre-OST0000 complete [ 9101.244035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.758906] LustreError: 338458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.828274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.865484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9106.920714] LustreError: 338459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.038506] LustreError: 338458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.296914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.552635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9117.356676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9122.594029] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9126.116492] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9132.899834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.338576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9139.557546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9144.659408] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9146.673979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9162.212832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.215530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9162.228467] Lustre: Skipped 3 previous similar messages [ 9162.240038] Lustre: Skipped 4 previous similar messages [ 9167.234648] Lustre: server umount lustre-MDT0000 complete [ 9167.329933] LustreError: 338459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9167.348143] LustreError: 338459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9170.704887] LustreError: 338441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878762 with bad export cookie 17020885309785124611 [ 9170.709506] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.713411] LustreError: 338441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9170.892616] Lustre: server umount lustre-MDT0001 complete [ 9185.041571] Lustre: server umount lustre-OST0000 complete [ 9197.664370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.948756] LustreError: 342028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.015198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.929113] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9209.179847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.631441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9217.720956] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9221.425838] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9228.315760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9236.232958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9241.719562] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9243.184616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9253.347886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9253.355495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.358343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9253.376033] Lustre: Skipped 1 previous similar message [ 9253.390041] Lustre: Skipped 1 previous similar message [ 9257.224686] Lustre: server umount lustre-OST0000 complete [ 9263.587597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9263.597768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9263.605461] Lustre: Skipped 1 previous similar message [ 9263.618516] Lustre: Skipped 1 previous similar message [ 9267.643652] Lustre: server umount lustre-MDT0000 complete [ 9268.705216] LustreError: 342024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.716310] LustreError: 342024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9270.744930] LustreError: 342011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878862 with bad export cookie 17020885309785125633 [ 9270.753373] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.755408] LustreError: 342011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9271.075566] Lustre: server umount lustre-MDT0001 complete [ 9279.921469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9282.085342] Key type lgssc unregistered [ 9282.312375] LNet: 345204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9282.317545] LNetError: 345204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9282.334610] LNet: Removed LNI 192.168.203.155@tcp [ 9282.952283] Key type .llcrypt unregistered [ 9282.954526] Key type ._llcrypt unregistered [ 9304.298104] Key type ._llcrypt registered [ 9304.300075] Key type .llcrypt registered [ 9304.376888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9305.050960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9305.129972] alg: No test for adler32 (adler32-zlib) [ 9306.141672] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 9306.343061] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9307.991647] Key type lgssc registered [ 9308.984541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9316.430900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9316.445284] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.452565] LustreError: 346625:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9316.490347] LustreError: 346625:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9316.505654] LustreError: 346625:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9316.513726] LustreError: 346625:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9316.521704] LustreError: 346625:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9323.432037] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.439972] LustreError: 347022:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9323.463089] LustreError: 347022:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9323.473664] LustreError: 347022:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9323.482906] LustreError: 347022:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9323.489714] LustreError: 347022:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9339.160220] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9341.249667] Key type lgssc unregistered [ 9341.491464] LNet: 348009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9341.500915] LNetError: 348009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9341.524690] LNet: Removed LNI 192.168.203.155@tcp [ 9342.156163] Key type .llcrypt unregistered [ 9342.159280] Key type ._llcrypt unregistered [ 9356.714967] Key type ._llcrypt registered [ 9356.718471] Key type .llcrypt registered [ 9356.806220] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9368.462287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9369.039093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9369.089715] alg: No test for adler32 (adler32-zlib) [ 9370.181739] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 9370.339683] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9372.023733] Key type lgssc registered [ 9373.036823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9405.682613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9405.722372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.040611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9407.065740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9407.114872] Lustre: lustre-MDT0000: new disk, initializing [ 9407.194287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.205244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9411.702293] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9422.643820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.756675] Lustre: 351111:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9422.801325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9422.805880] Lustre: Skipped 1 previous similar message [ 9422.864617] Lustre: lustre-MDT0001: new disk, initializing [ 9422.988741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.024981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9423.037963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9427.048851] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9434.034824] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9437.511230] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9443.899492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9444.168427] Lustre: lustre-OST0000: new disk, initializing [ 9444.171125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9444.181892] Lustre: 352851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9444.330836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.065621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9450.081717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9450.165428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9450.285391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9457.344410] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9461.144109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9461.331572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9465.140175] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9465.366255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9470.946437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9470.959945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.962284] LustreError: Skipped 1 previous similar message [ 9470.966186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9470.991808] Lustre: Skipped 1 previous similar message [ 9473.388588] Lustre: server umount lustre-OST0000 complete [ 9479.138535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9479.148911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9479.168274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9479.171369] Lustre: Skipped 1 previous similar message [ 9481.184386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.185559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.653768] Lustre: server umount lustre-MDT0000 complete [ 9486.178937] LustreError: 351888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879077 with bad export cookie 16845971732076697008 [ 9486.190105] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9486.198703] LustreError: 351888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9486.509783] Lustre: server umount lustre-MDT0001 complete [ 9497.138121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9499.308505] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:51:29 (1779879089) [ 9507.121151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.721341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.938399] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9512.931104] LustreError: 355165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.968855] LustreError: 355165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9518.054764] LustreError: 355164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.542440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.837044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9525.409130] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9530.295636] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9534.205615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9540.164327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9540.467742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.691896] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9548.939967] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9553.494935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9553.619000] Lustre: lustre-OST0001: new disk, initializing [ 9553.623606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9553.631807] Lustre: 357658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9557.900474] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9560.127879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9560.138662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9560.177430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9564.339836] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9570.276840] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.295366] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9570.299209] Lustre: Skipped 1 previous similar message [ 9573.797451] Lustre: server umount lustre-OST0001 complete [ 9575.397672] LustreError: 358070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9575.414414] LustreError: 358070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9580.529334] LustreError: 356757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.558449] LustreError: 356757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9585.636739] LustreError: 356758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.657768] LustreError: 356758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9590.757565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.764567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9590.768646] Lustre: Skipped 2 previous similar messages [ 9590.780613] Lustre: Skipped 2 previous similar messages [ 9595.881433] LustreError: 356758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.906289] LustreError: 356758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9600.994834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9601.001761] Lustre: Skipped 3 previous similar messages [ 9603.039121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9603.163954] Lustre: server umount lustre-OST0000 complete [ 9609.184864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9609.191548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9612.838685] Lustre: server umount lustre-MDT0000 complete [ 9616.354408] LustreError: 356225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.389397] LustreError: 356225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9616.411150] LustreError: 355146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879207 with bad export cookie 16845971732076697890 [ 9616.421121] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9616.425960] LustreError: 355146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9622.767133] Lustre: server umount lustre-MDT0001 complete [ 9635.024531] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9637.934259] Key type lgssc unregistered [ 9638.355070] LNet: 359518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9638.373994] LNetError: 359518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9638.393194] LNet: Removed LNI 192.168.203.155@tcp [ 9639.477967] Key type .llcrypt unregistered [ 9639.481913] Key type ._llcrypt unregistered [ 9687.061808] Key type ._llcrypt registered [ 9687.065342] Key type .llcrypt registered [ 9687.215791] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9688.565428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9688.649197] alg: No test for adler32 (adler32-zlib) [ 9689.825247] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 9690.205213] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9691.999236] Key type lgssc registered [ 9693.451567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9702.139309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9702.174191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.582532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9703.627524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9703.969982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9708.308987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9716.509859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.689616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9716.725339] Lustre: 362299:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9716.892519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9720.883315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9726.155390] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9729.414598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9732.067878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.082167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.088144] Lustre: Skipped 1 previous similar message [ 9737.188706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9737.199705] Lustre: Skipped 1 previous similar message [ 9738.084965] Lustre: server umount lustre-MDT0000 complete [ 9741.743616] LustreError: 363026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879333 with bad export cookie 4364308217003045093 [ 9741.745502] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9741.758055] LustreError: 363026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9742.147732] Lustre: server umount lustre-MDT0001 complete [ 9751.496440] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:55:42 (1779879342) [ 9757.979097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.508539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.948552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9763.823633] LustreError: 364588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.854419] LustreError: 364588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9768.930609] LustreError: 364589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9771.358395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9771.582829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9775.350724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9780.661720] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9784.372084] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9790.944892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9791.093561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9791.304176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.608956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9802.905001] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9811.220723] Lustre: Permanently deactivating lustre-OST0000 [ 9811.236208] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9818.117434] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9821.179677] LustreError: 367234:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9821.196288] LustreError: 367234:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9821.208297] LustreError: 367234:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9825.018558] LustreError: 367282:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9825.024132] LustreError: 367282:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9833.400540] Lustre: server umount lustre-OST0000 complete [ 9836.073986] LustreError: 367532:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9836.080239] LustreError: 367532:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9841.633219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.656859] Lustre: Skipped 1 previous similar message [ 9841.668425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9846.753256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9846.758854] Lustre: Skipped 2 previous similar messages [ 9847.915882] Lustre: server umount lustre-MDT0000 complete [ 9851.872138] LustreError: 364588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.908858] LustreError: 364588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9852.233478] LustreError: 366938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879443 with bad export cookie 4364308217003045674 [ 9852.235127] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9852.246166] LustreError: 366938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9861.844988] Lustre: server umount lustre-MDT0001 complete [ 9871.994454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.439751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9879.461602] LustreError: 368673:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9885.421614] Lustre: 369013:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [ 9888.010494] Lustre: server umount MGS complete [ 9899.543056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.980511] LustreError: 369739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.999884] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9900.013915] Lustre: Skipped 1 previous similar message [ 9900.058631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9904.333790] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9905.120801] LustreError: 369740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.156815] LustreError: 369740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9912.340047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.807794] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9917.567155] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9924.366660] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9929.421368] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9931.655088] Lustre: Permanently reactivating lustre-OST0000 [ 9931.672808] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9931.679419] Lustre: Skipped 2 previous similar messages [ 9961.439958] LustreError: 371340:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9969.261706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9969.555627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9969.560264] Lustre: Skipped 1 previous similar message [ 9975.498033] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9981.046449] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9987.050385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9987.052842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.058888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9987.107869] Lustre: Skipped 2 previous similar messages [ 9992.426964] Lustre: server umount lustre-OST0000 complete [ 9997.281173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.287848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.304873] Lustre: Skipped 1 previous similar message [ 9997.309962] Lustre: Skipped 3 previous similar messages [10002.340990] Lustre: server umount lustre-MDT0000 complete [10002.401576] LustreError: 371678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.414638] LustreError: 371678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10006.034707] LustreError: 369721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879597 with bad export cookie 4364308217003046717 [10006.039717] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.044411] LustreError: 369721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10006.387987] Lustre: server umount lustre-MDT0001 complete [10017.586896] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10020.656193] Key type lgssc unregistered [10021.040534] LNet: 373843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10021.055859] LNetError: 373843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10021.067937] LNet: Removed LNI 192.168.203.155@tcp [10021.800658] Key type .llcrypt unregistered [10021.803555] Key type ._llcrypt unregistered [10040.874907] Key type ._llcrypt registered [10040.879031] Key type .llcrypt registered [10040.976662] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10054.264876] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10055.182233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10055.238304] alg: No test for adler32 (adler32-zlib) [10056.266129] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10056.481986] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10058.152762] Key type lgssc registered [10059.064535] Lustre: Echo OBD driver; http://www.lustre.org/ [10094.226695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10094.261817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.539209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10095.574123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10095.671308] Lustre: lustre-MDT0000: new disk, initializing [10095.757233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10095.776232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10100.076135] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10111.486932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.607195] Lustre: 376971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10111.653313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10111.658491] Lustre: Skipped 1 previous similar message [10111.723216] Lustre: lustre-MDT0001: new disk, initializing [10111.835628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10111.882849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10111.890517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10116.083470] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10123.601072] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10127.368394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10133.375513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10133.581799] Lustre: lustre-OST0000: new disk, initializing [10133.583449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10133.586272] Lustre: 378715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10133.638520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10138.401315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10142.791969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10142.801299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10142.897629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10144.959605] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10149.562824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10149.725541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10153.348149] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10153.551813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10158.564142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.566400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10158.570672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.585954] Lustre: Skipped 1 previous similar message [10158.592175] LustreError: Skipped 1 previous similar message [10161.326703] Lustre: server umount lustre-OST0000 complete [10168.290149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10168.301660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.327607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.334682] Lustre: Skipped 1 previous similar message [10173.926565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10173.932684] Lustre: Skipped 2 previous similar messages [10179.043637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10179.046331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [10179.056815] Lustre: Skipped 1 previous similar message [10179.275657] Lustre: server umount lustre-MDT0000 complete [10182.231594] LustreError: 376963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879773 with bad export cookie 13246845694762778474 [10182.232981] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.248920] LustreError: 376963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10182.569972] Lustre: server umount lustre-MDT0001 complete [10190.690684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10192.738397] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:03:03 (1779879783) [10200.168135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.733070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10205.173000] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10206.181931] LustreError: 381027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10206.204858] LustreError: 381027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10211.308700] LustreError: 381028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.916655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.267378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10218.436096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10224.566927] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10228.836308] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10236.367637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10236.773768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10243.646465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10250.776512] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10253.450838] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10256.551081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10264.546275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10264.561639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.576866] Lustre: Skipped 1 previous similar message [10264.586669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10266.601887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10266.618412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.073917] Lustre: server umount lustre-OST0000 complete [10275.810551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.824540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.842046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.847877] Lustre: Skipped 2 previous similar messages [10279.380581] Lustre: server umount lustre-MDT0000 complete [10281.959147] LustreError: 381022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.991845] LustreError: 381022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10283.591443] LustreError: 381009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879875 with bad export cookie 13246845694762779356 [10283.593611] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.605164] LustreError: 381009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10283.972467] Lustre: server umount lustre-MDT0001 complete [10294.162423] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10296.610640] Key type lgssc unregistered [10296.899368] LNet: 384395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10296.912053] LNetError: 384395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10296.928673] LNet: Removed LNI 192.168.203.155@tcp [10297.715762] Key type .llcrypt unregistered [10297.721674] Key type ._llcrypt unregistered [10322.087159] Key type ._llcrypt registered [10322.089753] Key type .llcrypt registered [10322.251777] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10322.527476] Key type .llcrypt unregistered [10322.531872] Key type ._llcrypt unregistered [10342.202387] Key type ._llcrypt registered [10342.204703] Key type .llcrypt registered [10342.346911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10342.990501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10343.054071] alg: No test for adler32 (adler32-zlib) [10344.088627] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10344.373609] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10346.135934] Key type lgssc registered [10347.051592] Lustre: Echo OBD driver; http://www.lustre.org/ [10355.666651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10355.716340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.220948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10361.167507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10366.506207] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.519364] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10370.215124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.594474] LustreError: 386582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.618963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10375.673686] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10375.715139] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10377.099031] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10380.791147] LustreError: 386582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10380.838196] LustreError: 386582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10381.824393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10383.513642] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10386.917271] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.943576] LustreError: 386581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10393.260398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.802473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10399.220180] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10409.440539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10409.446313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.456924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10409.468834] Lustre: Skipped 1 previous similar message [10410.804549] Lustre: server umount lustre-OST0000 complete [10416.104922] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10416.112209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10416.123820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10416.129640] Lustre: Skipped 1 previous similar message [10417.094641] Lustre: server umount lustre-MDT0000 complete [10419.680130] LustreError: 386582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.680321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.703223] LustreError: 386582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10420.676742] LustreError: 386562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880012 with bad export cookie 4073369532265794640 [10420.685600] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.691836] LustreError: 386562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10420.939807] Lustre: server umount lustre-MDT0001 complete [10432.795944] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10435.472139] Key type lgssc unregistered [10435.828503] LNet: 389563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10435.834148] LNetError: 389563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10435.868797] LNet: Removed LNI 192.168.203.155@tcp [10436.610995] Key type .llcrypt unregistered [10436.615784] Key type ._llcrypt unregistered [10462.519950] Key type ._llcrypt registered [10462.522675] Key type .llcrypt registered [10462.662554] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10463.793776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10463.819059] alg: No test for adler32 (adler32-zlib) [10464.887710] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10465.107338] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10466.808605] Key type lgssc registered [10467.799606] Lustre: Echo OBD driver; http://www.lustre.org/ [10475.424464] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10475.446992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.039820] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10496.159258] LustreError: 390937:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [10522.912411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10530.126302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10536.055775] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10541.678293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.984051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10542.059956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10545.736590] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10547.171740] LustreError: 390962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.183580] LustreError: 390962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10552.293740] LustreError: 390963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.998282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.381825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10557.605894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10563.375818] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10567.322777] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10573.794992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10573.811472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10573.828080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10574.817486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10574.820057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10575.495226] Lustre: server umount lustre-OST0000 complete [10579.936811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10579.964572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10579.972101] Lustre: Skipped 2 previous similar messages [10581.469211] Lustre: server umount lustre-MDT0000 complete [10585.057958] LustreError: 391865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.071988] LustreError: 391865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10585.228382] LustreError: 393212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880176 with bad export cookie 8688228576784100938 [10585.240291] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10585.243438] LustreError: 393212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10585.783200] Lustre: server umount lustre-MDT0001 complete [10596.260845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10599.025965] Key type lgssc unregistered [10599.318441] LNet: 394147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10599.332699] LNetError: 394147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10599.363228] LNet: Removed LNI 192.168.203.155@tcp [10600.145137] Key type .llcrypt unregistered [10600.146956] Key type ._llcrypt unregistered [10622.921331] Key type ._llcrypt registered [10622.926329] Key type .llcrypt registered [10623.061418] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10624.371192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10624.404683] alg: No test for adler32 (adler32-zlib) [10625.494301] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10625.751649] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10627.511203] Key type lgssc registered [10628.596686] Lustre: Echo OBD driver; http://www.lustre.org/ [10636.108957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10636.128275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.600654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10641.589919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10648.044651] LustreError: 395547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.061222] LustreError: 395547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10650.937414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.353433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10656.073579] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10662.094380] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10666.098104] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10673.492682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10673.989784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10679.695947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10683.205559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10685.964738] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10692.578063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10692.585424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10692.596910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10693.211375] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [10693.606826] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10693.624751] Lustre: Skipped 1 previous similar message [10695.985305] Lustre: server umount lustre-MDT0000 complete [10698.320251] LustreError: 395529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1779880289 with bad export cookie 8576783430757072469 [10698.723327] LustreError: 396520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10698.745931] LustreError: 396520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10700.278398] LustreError: 395528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880291 with bad export cookie 8576783430757071573 [10700.298799] LustreError: 395528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10700.308661] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10701.792484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10701.813134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10701.844971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10701.867891] Lustre: Skipped 3 previous similar messages [10703.841953] LustreError: 396260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.844217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10703.862608] LustreError: 396260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10705.889217] LustreError: 395548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10705.923285] LustreError: 395548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10706.580223] Lustre: server umount lustre-OST0000 complete [10709.985865] LustreError: 396260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.015749] LustreError: 396260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10714.372363] Lustre: server umount lustre-MDT0001 complete [10727.610072] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10730.445545] Key type lgssc unregistered [10730.828919] LNet: 398876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10730.843063] LNetError: 398876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10730.871928] LNet: Removed LNI 192.168.203.155@tcp [10731.908244] Key type .llcrypt unregistered [10731.911222] Key type ._llcrypt unregistered [10757.398868] Key type ._llcrypt registered [10757.401992] Key type .llcrypt registered [10757.559689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10758.788967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10758.812876] alg: No test for adler32 (adler32-zlib) [10759.978130] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10760.251473] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10761.919207] Key type lgssc registered [10762.986837] Lustre: Echo OBD driver; http://www.lustre.org/ [10770.913648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10770.941787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.449811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.866259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10781.667148] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.700735] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10785.442401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.892380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10790.275764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10796.279848] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10800.653845] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10808.301595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10808.663229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10815.221309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10816.954613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10821.294829] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10827.233406] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10827.237296] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10827.254579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10827.265274] Lustre: Skipped 1 previous similar message [10827.273716] Lustre: Skipped 1 previous similar message [10829.423056] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [10832.363237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10832.375033] Lustre: Skipped 1 previous similar message [10832.990934] Lustre: server umount lustre-MDT0001 complete [10834.507047] LustreError: 400272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.484061] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.516638] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10842.592514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10842.605481] LustreError: 400277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.606146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10842.608747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10846.187689] Lustre: server umount lustre-OST0000 complete [10847.717235] LustreError: 400278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.441118] Lustre: server umount lustre-MDT0000 complete [10865.669916] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10868.821457] Key type lgssc unregistered [10869.215773] LNet: 403603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10869.228468] LNetError: 403603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10869.254737] LNet: Removed LNI 192.168.203.155@tcp [10870.300828] Key type .llcrypt unregistered [10870.304214] Key type ._llcrypt unregistered [10899.092580] Key type ._llcrypt registered [10899.094250] Key type .llcrypt registered [10899.227453] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10913.847251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10915.017174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10915.065150] alg: No test for adler32 (adler32-zlib) [10916.129622] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10916.403266] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10918.135362] Key type lgssc registered [10919.196222] Lustre: Echo OBD driver; http://www.lustre.org/ [10957.790754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10974.107601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10974.161438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.523277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10975.548965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10975.613544] Lustre: lustre-MDT0000: new disk, initializing [10975.704543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10975.738166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10980.451514] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10995.522036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10995.972508] Lustre: lustre-OST0000: new disk, initializing [10995.977927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10995.985939] Lustre: Skipped 1 previous similar message [10995.991317] Lustre: 408573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10996.079609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11001.468974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11001.482702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11001.522357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11004.406967] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11013.672565] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11036.095614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11036.389744] Lustre: 407571:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11036.633976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11036.714404] Lustre: lustre-MDT0001: new disk, initializing [11036.895473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11036.994320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11037.013457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11042.648958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11051.900843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11052.109382] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11056.692535] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11056.914878] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11061.745653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11061.757217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11064.912111] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11066.851114] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11066.856454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11066.857604] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.020723] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11070.036495] Lustre: Skipped 1 previous similar message [11075.126752] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11075.139962] Lustre: Skipped 2 previous similar messages [11076.063076] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11076.451171] Lustre: server umount lustre-MDT0001 complete [11082.221146] LustreError: 407581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.253075] LustreError: 407581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11087.336344] LustreError: 407582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.369341] LustreError: 407582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11092.450895] LustreError: 407576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.472024] LustreError: 407576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11097.603538] LustreError: 407582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.606628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.658626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.378508] Lustre: server umount lustre-OST0000 complete [11102.692639] LustreError: 410163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.422958] Lustre: server umount lustre-MDT0000 complete [11125.597949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11128.466357] Key type lgssc unregistered [11128.806875] LNet: 412138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11128.813818] LNetError: 412138:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11128.830237] LNet: Removed LNI 192.168.203.155@tcp [11129.703912] Key type .llcrypt unregistered [11129.706404] Key type ._llcrypt unregistered [11169.292666] Key type ._llcrypt registered [11169.297574] Key type .llcrypt registered [11169.415828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11169.659972] Key type .llcrypt unregistered [11169.665633] Key type ._llcrypt unregistered [11187.114991] Key type ._llcrypt registered [11187.117373] Key type .llcrypt registered [11187.223845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11188.392905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11188.544517] alg: No test for adler32 (adler32-zlib) [11189.687730] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [11189.959791] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11191.743161] Key type lgssc registered [11192.832237] Lustre: Echo OBD driver; http://www.lustre.org/ [11222.403943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11222.488275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.718035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11223.752054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11223.859446] Lustre: lustre-MDT0000: new disk, initializing [11223.955104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11224.018575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11229.551931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11240.943621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.076559] Lustre: 415499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11241.135988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11241.140493] Lustre: Skipped 1 previous similar message [11241.229074] Lustre: lustre-MDT0001: new disk, initializing [11241.334166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11241.360978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11241.374256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11245.834334] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11253.909972] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11258.515233] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11266.049473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11266.409929] Lustre: lustre-OST0000: new disk, initializing [11266.414380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11266.426928] Lustre: 417240:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11266.526947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11272.827786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11272.849324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11273.025869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11273.972500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11283.454440] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11292.641965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11292.649511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.668098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.665458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.666830] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.692780] Lustre: Skipped 1 previous similar message [11296.678315] Lustre: server umount lustre-MDT0000 complete [11300.668367] LustreError: 416278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880892 with bad export cookie 1721005892989267541 [11300.676483] LustreError: 416278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11300.676490] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11301.110445] Lustre: server umount lustre-MDT0001 complete [11315.622957] Lustre: server umount lustre-OST0000 complete [11328.650776] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:21:58 (1779880918) [11336.471484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.031284] LustreError: 419161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.051883] LustreError: 419161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11337.126635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11341.801276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11342.307372] LustreError: 419162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.434636] LustreError: 419161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.871765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11352.447343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11357.033106] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11363.893531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11368.301986] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11388.740147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.286468] Lustre: Found index 0 for lustre-OST0000, updating log [11389.663704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11396.955772] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11403.813404] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11408.112692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11417.572959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11417.588071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.616304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11417.620629] Lustre: Skipped 2 previous similar messages [11420.910108] Lustre: server umount lustre-OST0000 complete [11437.142732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.156214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11437.164439] Lustre: Skipped 2 previous similar messages [11437.191222] Lustre: Skipped 4 previous similar messages [11442.144447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11442.154764] Lustre: Skipped 1 previous similar message [11443.647211] Lustre: server umount lustre-MDT0000 complete [11445.224447] LustreError: 419161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11445.238470] LustreError: 419161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11448.049658] LustreError: 419142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779881039 with bad export cookie 1721005892989268934 [11448.051650] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11448.059803] LustreError: 419142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11448.507644] Lustre: server umount lustre-MDT0001 complete [11452.256190] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [11459.598594] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:24:09 (1779881049) [12082.352499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.781576] LustreError: 425234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.800086] LustreError: 425234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12082.856475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12087.219697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12088.304978] LustreError: 425235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.409343] LustreError: 425234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.970224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.336261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12100.994420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12105.916705] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12110.156243] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12130.227408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12130.416832] Lustre: Found index 0 for lustre-OST0000, updating log [12130.703031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12131.780960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12136.508147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12141.797702] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12143.457536] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [12144.612429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.623485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.873362] Lustre: server umount lustre-OST0000 complete [12154.785330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.791888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12154.808962] Lustre: Skipped 1 previous similar message [12156.898456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.910652] Lustre: Skipped 4 previous similar messages [12160.993920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.006205] Lustre: Skipped 7 previous similar messages [12168.159162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12168.292127] LustreError: 425228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.313043] LustreError: 425228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12168.440460] Lustre: server umount lustre-MDT0000 complete [12172.385725] LustreError: 425231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.410951] LustreError: 425231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12172.630520] LustreError: 425216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779881764 with bad export cookie 1721005892989270019 [12172.631279] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.642168] LustreError: 425216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12173.368723] Lustre: server umount lustre-MDT0001 complete [12204.455672] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12215.115918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12248.041645] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:37:18 (1779881838) [12264.028813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12266.377946] Key type lgssc unregistered [12266.641535] LNet: 431460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12266.646943] LNetError: 431460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12391.591264] LNet: Removed LNI 192.168.203.155@tcp [12392.350885] Key type .llcrypt unregistered [12392.354110] Key type ._llcrypt unregistered [12407.844108] Key type ._llcrypt registered [12407.846483] Key type .llcrypt registered [12407.928093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12408.651685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12408.669489] alg: No test for adler32 (adler32-zlib) [12409.713501] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [12409.906977] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12411.567196] Key type lgssc registered [12412.520779] Lustre: Echo OBD driver; http://www.lustre.org/ [12415.785691] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [12416.324052] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [12417.017881] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [12417.597716] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [12418.162477] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [12418.781385] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [12419.416230] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [12420.100667] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [12420.682286] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [12421.313724] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [12421.928172] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [12422.500324] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [12423.053651] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [12423.752549] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [12424.424672] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [12425.212608] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [12425.930984] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [12426.632974] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [12427.459043] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [12428.108976] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [12428.804106] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [12429.601570] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [12430.425577] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [12431.198963] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [12431.969039] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [12432.697609] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [12433.607884] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [12434.406681] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [12435.281748] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [12436.021608] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [12436.871175] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [12437.858995] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [12438.960671] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [12439.799724] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [12440.750554] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [12441.744852] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [12448.118146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12448.162333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.301862] Lustre: 434475:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [12449.389300] LNetError: 431937:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12449.405666] LNetError: Protocol error connecting to 192.168.203.155@tcp119 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12449.422399] LNetError: Skipped 8 previous similar messages [12449.540443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12449.588279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12449.685761] Lustre: lustre-MDT0000: new disk, initializing [12449.795050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12449.810107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12449.887110] LNet: 7 local NIs in recovery (showing 7): 192.168.203.155@tcp122, 192.168.203.155@tcp124, 192.168.203.155@tcp125, 192.168.203.155@tcp126, 192.168.203.155@tcp127, 192.168.203.155@tcp128, 192.168.203.155@tcp129 [12450.468464] LNetError: 434509:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12450.489948] LNetError: 434509:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [12450.500074] LNetError: Protocol error connecting to 192.168.203.155@tcp103 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12450.508912] LNetError: Skipped 1 previous similar message [12453.027176] LNetError: 431937:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12453.034534] LNetError: 431937:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [12453.043970] LNetError: Protocol error connecting to 192.168.203.155@tcp125 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12453.056791] LNetError: Skipped 9 previous similar messages [12453.922963] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12457.123517] LNetError: 434501:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12457.136708] LNetError: 434501:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [12457.146177] LNetError: Protocol error connecting to 192.168.203.155@tcp125 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12457.157984] LNetError: Skipped 4 previous similar messages [12463.689907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.858595] Lustre: 434494:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12463.953627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12463.956925] Lustre: Skipped 1 previous similar message [12464.025533] Lustre: lustre-MDT0001: new disk, initializing [12464.047215] Lustre: 435277:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [12464.052625] Lustre: 435277:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12464.132542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12464.151638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12464.165734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12465.313906] LNetError: 434503:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12465.321618] LNetError: Protocol error connecting to 192.168.203.155@tcp125 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12468.649986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12470.855226] Lustre: 435623:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [12470.865258] Lustre: 435623:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12476.513535] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12479.525021] LNetError: 431938:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12479.532224] LNetError: Protocol error connecting to 192.168.203.155@tcp103 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12480.817088] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12488.661518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12488.889440] Lustre: lustre-OST0000: new disk, initializing [12488.896804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12488.903534] Lustre: 436240:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12489.006681] Lustre: 436245:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [12489.019567] Lustre: 436245:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12489.082465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12494.981522] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12495.910978] LNetError: 431937:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12495.919789] LNetError: 431937:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [12495.926097] LNetError: Protocol error connecting to 192.168.203.155@tcp122 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12495.939796] LNetError: Skipped 12 previous similar messages [12496.409493] Lustre: 436659:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [12496.418423] Lustre: 436659:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12496.485367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12496.513976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12496.639696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12501.807303] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12510.368659] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp125 [12512.229964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.233145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12512.237497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12512.255154] Lustre: Skipped 1 previous similar message [12514.403914] Lustre: server umount lustre-OST0000 complete [12520.417259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12520.426690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.440882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12520.445856] Lustre: Skipped 1 previous similar message [12522.464524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12522.464550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12522.487539] Lustre: Skipped 1 previous similar message [12523.836814] Lustre: server umount lustre-MDT0000 complete [12526.504866] LustreError: 436239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779882118 with bad export cookie 5087913502268577011 [12526.506551] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12526.517349] LustreError: 436239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12526.739286] Lustre: server umount lustre-MDT0001 complete [12528.673644] LNetError: 434503:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [12528.685296] LNetError: 434503:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 23 previous similar messages [12528.693623] LNetError: Protocol error connecting to 192.168.203.155@tcp121 at host 192.168.203.155:988: is it running a compatible version of Lustre? [12528.706542] LNetError: Skipped 23 previous similar messages [12535.408379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12537.543332] Key type lgssc unregistered [12537.830556] LNet: 438024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12537.841540] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12537.847054] LNet: Removed LNI 192.168.203.155@tcp [12538.911612] LNet: Removed LNI 192.168.203.155@tcp100 [12538.917656] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12538.926042] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12538.932270] LNet: Removed LNI 192.168.203.155@tcp101 [12540.000020] LNet: Removed LNI 192.168.203.155@tcp102 [12540.006351] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12540.012922] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12540.021737] LNet: Removed LNI 192.168.203.155@tcp103 [12541.087228] LNet: Removed LNI 192.168.203.155@tcp104 [12542.114383] LNet: Removed LNI 192.168.203.155@tcp105 [12542.119108] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12542.131701] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12543.199725] LNet: Removed LNI 192.168.203.155@tcp106 [12544.223739] LNet: Removed LNI 192.168.203.155@tcp107 [12545.247978] LNet: Removed LNI 192.168.203.155@tcp108 [12546.335295] LNet: Removed LNI 192.168.203.155@tcp109 [12546.337394] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12546.348463] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [12546.355693] LNet: Removed LNI 192.168.203.155@tcp110 [12547.423311] LNet: Removed LNI 192.168.203.155@tcp111 [12548.447542] LNet: Removed LNI 192.168.203.155@tcp112 [12549.472624] LNet: Removed LNI 192.168.203.155@tcp113 [12550.495787] LNet: Removed LNI 192.168.203.155@tcp114 [12551.520300] LNet: Removed LNI 192.168.203.155@tcp115 [12552.543261] LNet: Removed LNI 192.168.203.155@tcp116 [12553.567374] LNet: Removed LNI 192.168.203.155@tcp117 [12554.655156] LNet: Removed LNI 192.168.203.155@tcp118 [12554.659915] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12554.673124] LNetError: 438024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12554.681886] LNet: Removed LNI 192.168.203.155@tcp119 [12555.743281] LNet: Removed LNI 192.168.203.155@tcp120 [12556.768341] LNet: Removed LNI 192.168.203.155@tcp121 [12557.793766] LNet: Removed LNI 192.168.203.155@tcp122 [12558.815935] LNet: Removed LNI 192.168.203.155@tcp123 [12559.839376] LNet: Removed LNI 192.168.203.155@tcp124 [12559.846117] LNet: Removed LNI 192.168.203.155@tcp125 [12560.864618] LNet: Removed LNI 192.168.203.155@tcp126 [12561.887427] LNet: Removed LNI 192.168.203.155@tcp127 [12562.911161] LNet: Removed LNI 192.168.203.155@tcp128 [12563.935745] LNet: Removed LNI 192.168.203.155@tcp129 [12563.944318] LNet: Removed LNI 192.168.203.155@tcp130 [12563.957245] LNet: Removed LNI 192.168.203.155@tcp131 [12563.966928] LNet: Removed LNI 192.168.203.155@tcp132 [12563.975757] LNet: Removed LNI 192.168.203.155@tcp133 [12563.979937] LNet: Removed LNI 192.168.203.155@tcp134 [12563.999844] LNet: Removed LNI 192.168.203.155@tcp135 [12564.798882] Key type .llcrypt unregistered [12564.801625] Key type ._llcrypt unregistered [12587.936405] Key type ._llcrypt registered [12587.940477] Key type .llcrypt registered [12588.055649] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12602.756500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12603.453821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12603.496068] alg: No test for adler32 (adler32-zlib) [12604.558278] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [12604.732170] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12606.391158] Key type lgssc registered [12607.571745] Lustre: Echo OBD driver; http://www.lustre.org/ [12641.594928] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:43:52 (1779882232) [12658.342911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12661.129847] Key type lgssc unregistered [12661.389355] LNet: 442061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12661.396767] LNetError: 442061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12661.419302] LNet: Removed LNI 192.168.203.155@tcp [12662.190074] Key type .llcrypt unregistered [12662.193640] Key type ._llcrypt unregistered [12677.298964] Key type ._llcrypt registered [12677.300036] Key type .llcrypt registered [12677.353289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12677.791649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12677.811180] alg: No test for adler32 (adler32-zlib) [12678.811547] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [12679.021873] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12680.663216] Key type lgssc registered [12681.678889] Lustre: Echo OBD driver; http://www.lustre.org/ [12691.764568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12691.813298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.094560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12693.135953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12693.187171] Lustre: lustre-MDT0000: new disk, initializing [12693.238489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12693.256576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12698.299094] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12708.701531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.856828] Lustre: 443273:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12708.910480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12708.915358] Lustre: Skipped 1 previous similar message [12708.988906] Lustre: lustre-MDT0001: new disk, initializing [12709.121386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12709.166291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12709.175541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12713.175238] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12720.693987] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12725.587902] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12733.163891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.497338] Lustre: lustre-OST0000: new disk, initializing [12733.501609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12733.509258] Lustre: 445018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12733.585368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12740.112289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12740.714309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12740.735921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12740.847414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12747.407356] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12751.446441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [12758.071754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12758.103805] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [12764.339069] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12764.384287] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [12768.183908] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12768.219925] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [12773.114203] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12773.149068] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [12778.328093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12778.346969] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [12784.053164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12784.117862] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [12789.679291] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12789.711552] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [12796.107356] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12796.134954] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [12801.130937] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12801.181884] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [12805.894764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12805.937716] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [12810.705441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12810.730605] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [12815.822888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12815.853065] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [12821.672128] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12821.697954] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [12828.339824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12828.363686] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [12833.950823] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12833.985269] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [12840.270949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12840.307273] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [12845.436225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12845.467989] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [12851.105407] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12851.134878] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [12857.274121] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12857.322622] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [12863.465584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12863.496137] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [12869.360576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12869.388231] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [12874.343560] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12874.368897] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [12880.016345] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12880.038830] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [12886.976964] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12887.022473] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [12893.067689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12893.092611] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [12901.222600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12901.244793] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [12907.190720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12907.207805] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [12913.366865] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12913.419397] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [12919.538582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12919.562079] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [12926.478651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12926.504337] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [12933.543621] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12933.581969] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [12940.923109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12940.973045] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [12947.265333] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12947.287027] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [12953.916504] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12953.942392] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [12959.942372] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12959.981807] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [12968.395929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12968.446634] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [12977.957235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12977.999286] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [12986.441797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12986.470812] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [12994.761991] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12994.810485] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [13003.473187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13003.503688] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [13011.758582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13011.786888] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [13031.904238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13031.922450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13031.939796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13032.929027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.932226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13032.979591] Lustre: Skipped 1 previous similar message [13032.994934] Lustre: Skipped 2 previous similar messages [13038.633124] Lustre: server umount lustre-MDT0000 complete [13042.656049] LustreError: 443264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779882634 with bad export cookie 12492645641442528813 [13042.659812] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13042.671043] LustreError: 443264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13043.209489] Lustre: server umount lustre-MDT0001 complete [13058.529108] Lustre: 442546:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779882634/real 1779882634] req@00000000377bf42a x1866341831314432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779882650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13058.567647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13062.624229] Lustre: 442547:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779882638/real 1779882638] req@00000000cd511ec0 x1866341831314560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779882654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13063.466559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13063.647148] Lustre: 442545:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779882639/real 1779882639] req@000000008d086849 x1866341831314688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779882655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13068.768275] Lustre: 442544:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779882644/real 1779882644] req@000000005e0fc4f6 x1866341831314816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779882660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13072.466234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13074.911286] Lustre: 442547:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779882650/real 1779882650] req@00000000010fe22b x1866341831315072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779882666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13088.453711] LustreError: 445011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13088.468447] LustreError: 445011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13088.492791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13089.510750] LustreError: 445012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.606977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13093.548422] LustreError: 445011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.976737] LustreError: 445012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13100.932447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13101.203546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13104.297389] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13105.873353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13111.794415] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13112.562609] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13115.607137] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13134.816525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13134.826959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13134.852064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13137.377645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13137.408424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13140.907137] Lustre: server umount lustre-OST0000 complete [13147.105857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13147.126414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13147.160243] Lustre: Skipped 1 previous similar message [13147.172726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13147.184765] Lustre: Skipped 2 previous similar messages [13151.045627] Lustre: server umount lustre-MDT0000 complete [13153.765066] LustreError: 468413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13153.805552] LustreError: 468413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13155.413825] LustreError: 443264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779882746 with bad export cookie 12492645641442531473 [13155.421964] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13155.424957] LustreError: 443264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13155.757971] Lustre: server umount lustre-MDT0001 complete [13167.647747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13170.026865] Key type lgssc unregistered [13170.423971] LNet: 470973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13170.431190] LNetError: 470973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13170.442296] LNet: Removed LNI 192.168.203.155@tcp [13170.444031] LNet: Removed LNI 192.168.203.155@tcp10 [13170.448863] LNet: Removed LNI 192.168.203.155@tcp11 [13170.451880] LNet: Removed LNI 192.168.203.155@tcp12 [13170.454799] LNet: Removed LNI 192.168.203.155@tcp13 [13170.456557] LNet: Removed LNI 192.168.203.155@tcp14 [13170.459267] LNet: Removed LNI 192.168.203.155@tcp15 [13170.462670] LNet: Removed LNI 192.168.203.155@tcp16 [13170.466147] LNet: Removed LNI 192.168.203.155@tcp17 [13170.469257] LNet: Removed LNI 192.168.203.155@tcp18 [13170.472296] LNet: Removed LNI 192.168.203.155@tcp19 [13170.474668] LNet: Removed LNI 192.168.203.155@tcp20 [13170.477630] LNet: Removed LNI 192.168.203.155@tcp21 [13170.480472] LNet: Removed LNI 192.168.203.155@tcp22 [13170.483640] LNet: Removed LNI 192.168.203.155@tcp23 [13170.486507] LNet: Removed LNI 192.168.203.155@tcp24 [13170.489266] LNet: Removed LNI 192.168.203.155@tcp25 [13170.491759] LNet: Removed LNI 192.168.203.155@tcp26 [13170.494137] LNet: Removed LNI 192.168.203.155@tcp27 [13170.497246] LNet: Removed LNI 192.168.203.155@tcp28 [13170.500307] LNet: Removed LNI 192.168.203.155@tcp29 [13170.503380] LNet: Removed LNI 192.168.203.155@tcp30 [13170.506298] LNet: Removed LNI 192.168.203.155@tcp31 [13170.509260] LNet: Removed LNI 192.168.203.155@tcp32 [13170.512099] LNet: Removed LNI 192.168.203.155@tcp33 [13170.514864] LNet: Removed LNI 192.168.203.155@tcp34 [13170.517558] LNet: Removed LNI 192.168.203.155@tcp35 [13170.520425] LNet: Removed LNI 192.168.203.155@tcp36 [13170.523264] LNet: Removed LNI 192.168.203.155@tcp37 [13170.526208] LNet: Removed LNI 192.168.203.155@tcp38 [13170.528345] LNet: Removed LNI 192.168.203.155@tcp39 [13170.531567] LNet: Removed LNI 192.168.203.155@tcp40 [13170.535362] LNet: Removed LNI 192.168.203.155@tcp41 [13170.538568] LNet: Removed LNI 192.168.203.155@tcp42 [13170.545047] LNet: Removed LNI 192.168.203.155@tcp43 [13170.549762] LNet: Removed LNI 192.168.203.155@tcp44 [13170.556474] LNet: Removed LNI 192.168.203.155@tcp45 [13170.559602] LNet: Removed LNI 192.168.203.155@tcp46 [13170.562382] LNet: Removed LNI 192.168.203.155@tcp47 [13170.564916] LNet: Removed LNI 192.168.203.155@tcp48 [13170.568496] LNet: Removed LNI 192.168.203.155@tcp49 [13170.582812] LNet: Removed LNI 192.168.203.155@tcp50 [13171.307694] Key type .llcrypt unregistered [13171.310968] Key type ._llcrypt unregistered [13190.872359] Key type ._llcrypt registered [13190.875324] Key type .llcrypt registered [13190.946835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13205.679802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13206.431567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13206.469176] alg: No test for adler32 (adler32-zlib) [13207.706236] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [13207.926980] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13209.591770] Key type lgssc registered [13210.800568] Lustre: Echo OBD driver; http://www.lustre.org/ [13247.933225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13247.970885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13249.339526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13249.385115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13249.603672] Lustre: lustre-MDT0000: new disk, initializing [13249.755987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13249.773679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13254.472600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13267.373787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13267.530746] Lustre: 474136:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13267.608964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13267.613229] Lustre: Skipped 1 previous similar message [13267.697668] Lustre: lustre-MDT0001: new disk, initializing [13267.799593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13267.873336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13267.888795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13272.765088] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13280.877569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13285.890600] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13294.325351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13294.499126] Lustre: *** cfs_fail_loc=90f, val=0*** [13294.581647] Lustre: lustre-OST0000: new disk, initializing [13294.589770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13294.603457] Lustre: 475926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13294.711718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13301.209884] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13301.317857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13301.330474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13301.480425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13309.337974] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13322.211847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13322.212505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13322.213201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13322.255687] LustreError: Skipped 1 previous similar message [13324.264952] Lustre: server umount lustre-OST0000 complete [13329.377184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13329.385990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13329.396644] Lustre: Skipped 1 previous similar message [13329.401408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13329.405625] Lustre: Skipped 1 previous similar message [13332.448715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13332.457994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13334.479644] Lustre: server umount lustre-MDT0000 complete [13338.229138] LustreError: 474129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779882929 with bad export cookie 16557396471440505229 [13338.236859] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13338.243184] LustreError: 474129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13338.679852] Lustre: server umount lustre-MDT0001 complete [13349.258028] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13351.964344] Key type lgssc unregistered [13352.297775] LNet: 477703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13352.307875] LNetError: 477703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13352.333753] LNet: Removed LNI 192.168.203.155@tcp [13353.101135] Key type .llcrypt unregistered [13353.104273] Key type ._llcrypt unregistered [13373.625531] Key type ._llcrypt registered [13373.627748] Key type .llcrypt registered [13373.741801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13374.695996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13374.748215] alg: No test for adler32 (adler32-zlib) [13376.038510] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [13376.332926] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13378.039370] Key type lgssc registered [13379.257955] Lustre: Echo OBD driver; http://www.lustre.org/ [13388.187220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13388.206511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.666383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13394.320904] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13400.036187] LustreError: 478960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13400.053143] LustreError: 478960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13404.441116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13404.848865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13409.313302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13415.245748] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13419.701373] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13426.507819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13426.810047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13433.446730] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13439.670151] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13451.085625] loop0: detected capacity change from 0 to 204800000 [13451.279177] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13451.299723] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13451.327133] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13457.152140] loop0: detected capacity change from 0 to 204800000 [13457.318198] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13457.329467] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13457.340238] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13462.269907] loop0: detected capacity change from 0 to 204800000 [13468.053000] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13468.225635] Lustre: fs15246-OST0000: new disk, initializing [13468.235689] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13468.246061] Lustre: 482215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13468.337945] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13474.371607] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13483.357438] loop1: detected capacity change from 0 to 204800000 [13488.826730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.953503] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13489.005560] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13489.091612] Lustre: fs15246-MDT0000: new disk, initializing [13489.232972] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13489.250132] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13489.355298] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13490.404720] Lustre: 483321:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000043431715 x1866342562049536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779883097 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13493.471747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13499.493603] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13499.502964] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13499.518965] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13514.222616] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13514.239656] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13519.336935] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13520.130459] Lustre: server umount fs15246-MDT0000 complete [13524.459770] LustreError: 478955:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13529.449303] Lustre: server umount fs15246-OST0000 complete [13542.880191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13542.881483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13542.894736] LustreError: Skipped 1 previous similar message [13542.895582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13542.922565] Lustre: Skipped 1 previous similar message [13544.929700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13544.945379] Lustre: Skipped 2 previous similar messages [13546.500503] Lustre: server umount lustre-OST0000 complete [13550.050936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13550.066960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13550.079482] Lustre: Skipped 1 previous similar message [13560.288129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13560.296306] Lustre: Skipped 3 previous similar messages [13564.383390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13564.651223] Lustre: server umount lustre-MDT0000 complete [13565.413120] LustreError: 479670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13565.435177] LustreError: 479670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13568.158674] LustreError: 485650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779883159 with bad export cookie 16130116945426481943 [13568.160784] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13568.175098] LustreError: 485650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13568.471275] Lustre: server umount lustre-MDT0001 complete [13579.872700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13582.710302] Key type lgssc unregistered [13583.042577] LNet: 486386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13583.052601] LNetError: 486386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13583.065859] LNet: Removed LNI 192.168.203.155@tcp [13583.866505] Key type .llcrypt unregistered [13583.871953] Key type ._llcrypt unregistered [13599.818263] Key type ._llcrypt registered [13599.821522] Key type .llcrypt registered [13599.897470] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13600.726916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13600.747201] alg: No test for adler32 (adler32-zlib) [13601.808396] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [13602.046084] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13603.687218] Key type lgssc registered [13604.759436] Lustre: Echo OBD driver; http://www.lustre.org/ [13638.172852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13647.954628] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13685.394347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13685.444407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13686.775955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13686.828593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13686.876618] Lustre: lustre-MDT0000: new disk, initializing [13686.930976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13686.947951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13690.808717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13700.669598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13700.798416] Lustre: 490811:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13700.863323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13700.873160] Lustre: Skipped 1 previous similar message [13700.971567] Lustre: lustre-MDT0001: new disk, initializing [13701.134342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13701.187433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13701.204712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13705.667432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13713.303843] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13717.165378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13723.869047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13724.187445] Lustre: lustre-OST0000: new disk, initializing [13724.199859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13724.209809] Lustre: 492556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13724.345349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13731.888348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13733.066698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13733.091210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13733.226153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13740.407497] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13746.008920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13746.255619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13749.760851] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13749.917704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13753.843666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13753.844472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13753.848269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13753.850406] LustreError: Skipped 1 previous similar message [13753.878286] Lustre: Skipped 1 previous similar message [13757.725937] Lustre: server umount lustre-OST0000 complete [13762.531802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13762.534932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13762.542478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13762.546572] Lustre: Skipped 1 previous similar message [13764.064940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13764.067885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13769.184134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13769.199589] Lustre: Skipped 2 previous similar messages [13774.307593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13774.312783] Lustre: Skipped 1 previous similar message [13776.351635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13776.802152] Lustre: server umount lustre-MDT0000 complete [13780.397402] LustreError: 490804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779883371 with bad export cookie 3302542855649501364 [13780.405339] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13780.414089] LustreError: 490804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13780.793269] Lustre: server umount lustre-MDT0001 complete [13808.271819] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13818.780857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13854.754764] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13588 sec ========= 08:04:04 (1779883444) [13856.728510] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:04:07 (1779883447) === [13860.862762] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:04:11 (1779883451) === [13908.192522] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13911.023670] Key type lgssc unregistered [13911.242826] LNet: 499011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13911.245959] LNetError: 499011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13911.263416] LNet: Removed LNI 192.168.203.155@tcp [13912.075565] Key type .llcrypt unregistered [13912.077544] Key type ._llcrypt unregistered