[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467023489 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003081] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007952] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022294] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026799] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.030451] *** VALIDATE proc *** [ 0.032197] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034338] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.040878] debug: unmapping init [mem 0xffffffffb6c59000-0xffffffffb6c60fff] [ 0.042904] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043660] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046019] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051268] rcu: Hierarchical SRCU implementation. [ 0.053460] smp: Bringing up secondary CPUs ... [ 0.054648] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.058190] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131023] node 0 deferred pages initialised in 68ms [ 0.134010] devtmpfs: initialized [ 0.135219] x86/mm: Memory block size: 128MB [ 0.138095] gcov: version magic: 0x41383552 [ 0.140058] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145310] pinctrl core: initialized pinctrl subsystem [ 0.147171] [ 0.147551] ************************************************************* [ 0.150013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152010] ** ** [ 0.154018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.158010] ** This means that this kernel is built to expose internal ** [ 0.160010] ** IOMMU data structures, which may compromise security on ** [ 0.162009] ** your system. ** [ 0.164009] ** ** [ 0.166009] ** If you see this message and you are not debugging the ** [ 0.168011] ** kernel, report this immediately to your vendor! ** [ 0.170013] ** ** [ 0.172011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174011] ************************************************************* [ 0.177714] NET: Registered protocol family 16 [ 0.179438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189422] cpuidle: using governor menu [ 0.190700] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192491] PCI: Using configuration type 1 for base access [ 0.194129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203294] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209202] cryptd: max_cpu_qlen set to 1000 [ 0.213266] ACPI: Added _OSI(Module Device) [ 0.215005] ACPI: Added _OSI(Processor Device) [ 0.216016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223335] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229250] ACPI: Interpreter enabled [ 0.231051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232051] ACPI: Using IOAPIC for interrupt routing [ 0.234164] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251224] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253082] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255080] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263507] acpiphp: Slot [2] registered [ 0.265164] acpiphp: Slot [5] registered [ 0.267099] acpiphp: Slot [6] registered [ 0.268161] acpiphp: Slot [7] registered [ 0.269059] acpiphp: Slot [8] registered [ 0.271093] acpiphp: Slot [9] registered [ 0.272114] acpiphp: Slot [10] registered [ 0.274099] acpiphp: Slot [3] registered [ 0.275069] acpiphp: Slot [4] registered [ 0.277115] acpiphp: Slot [11] registered [ 0.278062] acpiphp: Slot [12] registered [ 0.280145] acpiphp: Slot [13] registered [ 0.282095] acpiphp: Slot [14] registered [ 0.284113] acpiphp: Slot [15] registered [ 0.285171] acpiphp: Slot [16] registered [ 0.287080] acpiphp: Slot [17] registered [ 0.288101] acpiphp: Slot [18] registered [ 0.289081] acpiphp: Slot [19] registered [ 0.290086] acpiphp: Slot [20] registered [ 0.292082] acpiphp: Slot [21] registered [ 0.293111] acpiphp: Slot [22] registered [ 0.295083] acpiphp: Slot [23] registered [ 0.296084] acpiphp: Slot [24] registered [ 0.298092] acpiphp: Slot [25] registered [ 0.299123] acpiphp: Slot [26] registered [ 0.301096] acpiphp: Slot [27] registered [ 0.302090] acpiphp: Slot [28] registered [ 0.304117] acpiphp: Slot [29] registered [ 0.305101] acpiphp: Slot [30] registered [ 0.306139] acpiphp: Slot [31] registered [ 0.308062] PCI host bridge to bus 0000:00 [ 0.309025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326914] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330183] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344252] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355625] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357965] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.363770] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388835] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.394388] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449479] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507278] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.526015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.565015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593337] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.608015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.639538] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.692178] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.695524] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.698537] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.701959] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.707323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715117] iommu: Default domain type: Passthrough [ 0.717778] SCSI subsystem initialized [ 0.719123] ACPI: bus type USB registered [ 0.720119] usbcore: registered new interface driver usbfs [ 0.722143] usbcore: registered new interface driver hub [ 0.724074] usbcore: registered new device driver usb [ 0.726153] pps_core: LinuxPPS API ver. 1 registered [ 0.728012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.732076] PTP clock support registered [ 0.734141] EDAC MC: Ver: 3.0.0 [ 0.736201] PCI: Using ACPI for IRQ routing [ 0.738986] NetLabel: Initializing [ 0.740018] NetLabel: domain hash size = 128 [ 0.742018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744117] NetLabel: unlabeled traffic allowed by default [ 0.747047] vgaarb: loaded [ 0.748247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.749009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.753205] clocksource: Switched to clocksource kvm-clock [ 0.857381] VFS: Disk quotas dquot_6.6.0 [ 0.860293] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.862846] *** VALIDATE ramfs *** [ 0.864143] *** VALIDATE hugetlbfs *** [ 0.866695] pnp: PnP ACPI init [ 0.869292] pnp: PnP ACPI: found 6 devices [ 0.889918] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.895506] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899361] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903084] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906860] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.910943] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.915515] NET: Registered protocol family 2 [ 0.918908] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.925750] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929929] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.935844] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939617] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943982] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.947578] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950832] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954237] NET: Registered protocol family 1 [ 0.957749] RPC: Registered named UNIX socket transport module. [ 0.960073] RPC: Registered udp transport module. [ 0.961994] RPC: Registered tcp transport module. [ 0.963805] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966398] NET: Registered protocol family 44 [ 0.968310] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.970636] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.972988] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975529] PCI: CLS 0 bytes, default 64 [ 0.977887] Unpacking initramfs... [ 2.419321] debug: unmapping init [mem 0xffff96dbfcc54000-0xffff96dbfffbffff] [ 2.423424] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.425754] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.428512] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.962558] Initialise system trusted keyrings [ 2.964469] Key type blacklist registered [ 2.968319] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.980993] zbud: loaded [ 2.985369] *** VALIDATE nfs *** [ 2.988605] *** VALIDATE nfs4 *** [ 2.990281] pstore: using deflate compression [ 2.993176] Platform Keyring initialized [ 3.239598] NET: Registered protocol family 38 [ 3.244811] Key type asymmetric registered [ 3.248742] Asymmetric key parser 'x509' registered [ 3.257603] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.266994] io scheduler mq-deadline registered [ 3.271882] io scheduler kyber registered [ 3.278955] io scheduler bfq registered [ 3.296875] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.308579] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.313274] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.317375] ACPI: Power Button [PWRF] [ 3.324867] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.339404] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.358426] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.369883] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.390756] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.422125] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.452115] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.458931] Non-volatile memory driver v1.3 [ 3.461430] Linux agpgart interface v0.103 [ 3.505565] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.510106] vda: detected capacity change from 0 to 75083776 [ 3.534210] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.538769] vdb: detected capacity change from 0 to 1073741824 [ 3.561327] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.565260] vdc: detected capacity change from 0 to 2621440000 [ 3.599607] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.609417] vdd: detected capacity change from 0 to 2621440000 [ 3.645531] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.660678] vde: detected capacity change from 0 to 4294967296 [ 3.713329] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.730613] vdf: detected capacity change from 0 to 4294967296 [ 3.747821] libphy: Fixed MDIO Bus: probed [ 3.769596] usbcore: registered new interface driver usbserial_generic [ 3.781405] usbserial: USB Serial support registered for generic [ 3.792287] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.803447] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.805353] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.814060] mousedev: PS/2 mouse device common for all mice [ 3.817483] rtc_cmos 00:05: RTC can wake from S4 [ 3.825479] rtc_cmos 00:05: registered as rtc0 [ 3.827415] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.829175] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.834955] intel_pstate: CPU model not supported [ 3.842951] hid: raw HID events driver (C) Jiri Kosina [ 3.852571] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.854978] usbcore: registered new interface driver usbhid [ 3.854987] usbhid: USB HID core driver [ 3.855180] drop_monitor: Initializing network drop monitor service [ 3.855323] Initializing XFRM netlink socket [ 3.857470] NET: Registered protocol family 10 [ 3.865274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.879445] Segment Routing with IPv6 [ 3.895966] NET: Registered protocol family 17 [ 3.900204] mpls_gso: MPLS GSO support [ 3.926590] RAS: Correctable Errors collector initialized. [ 3.934195] AVX version of gcm_enc/dec engaged. [ 3.943432] AES CTR mode by8 optimization enabled [ 4.240512] sched_clock: Marking stable (4240490287, 0)->(5135249024, -894758737) [ 4.246257] registered taskstats version 1 [ 4.253556] Loading compiled-in X.509 certificates [ 4.256686] zswap: loaded using pool lzo/zbud [ 4.295342] Key type big_key registered [ 4.312465] Key type encrypted registered [ 4.314239] ima: No TPM chip found, activating TPM-bypass! [ 4.316921] ima: Allocated hash algorithm: sha1 [ 4.319517] ima: No architecture policies found [ 4.321855] evm: Initialising EVM extended attributes: [ 4.324391] evm: security.selinux [ 4.325384] evm: security.ima [ 4.326456] evm: security.capability [ 4.328291] evm: HMAC attrs: 0x1 [ 4.331598] rtc_cmos 00:05: setting system clock to 2026-08-27 00:55:53 UTC (1787792153) [ 4.340774] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 4.345124] debug: unmapping init [mem 0xffffffffb6982000-0xffffffffb6c58fff] [ 4.364130] Write protecting the kernel read-only data: 28672k [ 4.374819] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 4.380717] debug: unmapping init [mem 0xffffffffb5914000-0xffffffffb59fffff] [ 4.459750] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.503533] systemd[1]: Detected virtualization kvm. [ 4.510522] systemd[1]: Detected architecture x86-64. [ 4.514812] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.566557] systemd[1]: No hostname configured. [ 4.571386] systemd[1]: Set hostname to . [ 4.577037] random: systemd: uninitialized urandom read (16 bytes read) [ 4.581760] systemd[1]: Initializing machine ID from random generator. [ 4.748113] random: ln: uninitialized urandom read (6 bytes read) [ 4.919219] random: systemd: uninitialized urandom read (16 bytes read) [ 4.922750] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.928229] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.934658] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.678497] device-mapper: uevent: version 1.0.3 [ 6.686393] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.412475] virtio_net virtio0 ens2: renamed from eth0 [ 8.552837] random: fast init done [ 8.944618] scsi host0: ata_piix [ 8.960562] scsi host1: ata_piix [ 8.965569] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.969263] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.866312] random: crng init done [ 13.870361] random: 7 urandom warning(s) missed due to ratelimiting [ 15.252717] dracut-initqueue[590]: RTNETLINK answers: File exists [ 15.587285] hrtimer: interrupt took 5184936 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.554662] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.434587] printk: systemd: 22 output lines suppressed due to ratelimiting [ 23.651772] SELinux: Disabled at runtime. [ 23.847205] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.869870] systemd[1]: Detected virtualization kvm. [ 23.873493] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.250719] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.258098] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.283469] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.292426] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.298290] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.313085] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.328329] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK [ 27.960458] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.854372] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.496708] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.632456] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.856275] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.984144] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.738032] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.660924] NFS: Registering the id_resolver key type [ 36.664582] Key type id_resolver registered [ 36.665978] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg638-server login: [ 99.356739] libcfs: loading out-of-tree module taints kernel. [ 99.414414] Key type ._llcrypt registered [ 99.418240] Key type .llcrypt registered [ 99.536015] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 114.447635] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 115.850303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 115.859550] alg: No test for adler32 (adler32-zlib) [ 117.301216] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 118.510469] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 120.247556] Key type lgssc registered [ 122.336921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.323777] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.927886] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 193.395329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.423936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.636367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.671178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.754736] Lustre: lustre-MDT0000: new disk, initializing [ 194.816412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.832539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.542145] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.211066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.293370] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.323385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.327396] Lustre: Skipped 1 previous similar message [ 209.409223] Lustre: lustre-MDT0001: new disk, initializing [ 209.463541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.483204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.501669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.035422] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.026969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.974849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.230974] Lustre: lustre-OST0000: new disk, initializing [ 225.233574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.238589] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.285558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.122685] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.580482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.601378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.713455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.058901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.157363] Lustre: lustre-OST0001: new disk, initializing [ 244.160714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.164754] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.215929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.339749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.345932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.407142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.820265] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.395531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.548760] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.893253] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.658552] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing yml_node [ 279.724225] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 281.947512] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 283.935552] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 285.235064] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 21:00:32 EDT 2026 [ 300.087616] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 301.388728] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.898403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.908051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.929138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.504041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.505736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.527721] Lustre: Skipped 3 previous similar messages [ 322.937207] Lustre: server umount lustre-MDT0000 complete [ 325.956313] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792475 with bad export cookie 11395906617429596009 [ 325.976092] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 325.983863] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.624274] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.636324] Lustre: Skipped 2 previous similar messages [ 326.641801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.646668] Lustre: Skipped 1 previous similar message [ 330.656115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.665490] Lustre: Skipped 1 previous similar message [ 332.317327] Lustre: server umount lustre-MDT0001 complete [ 335.549604] Lustre: server umount lustre-OST0000 complete [ 338.887445] Lustre: server umount lustre-OST0001 complete [ 344.901461] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 353.944397] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 384.343755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.559918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.587551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.664798] Lustre: lustre-MDT0000: new disk, initializing [ 384.733157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.752068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.193243] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 397.756464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.887388] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.946472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.950574] Lustre: Skipped 1 previous similar message [ 398.052474] Lustre: lustre-MDT0001: new disk, initializing [ 398.153670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.190592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.204535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.795430] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 411.159764] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.512122] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 422.741552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.962682] Lustre: lustre-OST0000: new disk, initializing [ 422.970035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 422.977992] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.111383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.579689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.591758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 424.703070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 428.471116] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 437.305781] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 441.548919] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 441.776596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 446.504763] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 446.786448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 450.529185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.533343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.542888] LustreError: Skipped 1 previous similar message [ 450.550361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 450.584653] Lustre: Skipped 2 previous similar messages [ 450.620712] Lustre: Skipped 1 previous similar message [ 460.774845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.778348] Lustre: Skipped 2 previous similar messages [ 463.327345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.592866] Lustre: server umount lustre-OST0000 complete [ 469.984913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.004962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.788756] Lustre: server umount lustre-MDT0000 complete [ 476.128534] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.143840] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 477.396113] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792626 with bad export cookie 11395906617429597542 [ 477.397587] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.402907] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.869578] Lustre: server umount lustre-MDT0001 complete [ 482.188833] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:03:49 (1787792629) [ 485.302468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 486.778679] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:03:54 (1787792634) [ 493.448366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.998505] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.088460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.758205] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 499.170219] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.197522] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 504.291027] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.985034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.056774] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 524.898961] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.974776] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.122460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.081217] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 552.977725] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 564.211183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.221477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.241650] Lustre: Skipped 1 previous similar message [ 564.248843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.253919] Lustre: Skipped 3 previous similar messages [ 565.368441] Lustre: server umount lustre-OST0000 complete [ 575.415340] Lustre: server umount lustre-MDT0000 complete [ 579.108342] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792728 with bad export cookie 11395906617429598424 [ 579.130681] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.451288] Lustre: server umount lustre-MDT0001 complete [ 589.831097] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 592.822400] Key type lgssc unregistered [ 593.170030] LNet: 22993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.179694] LNetError: 22993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.218913] LNet: Removed LNI 192.168.206.138@tcp [ 595.293225] Key type .llcrypt unregistered [ 595.296820] Key type ._llcrypt unregistered [ 616.588306] Key type ._llcrypt registered [ 616.592524] Key type .llcrypt registered [ 616.685726] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 617.498759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.525123] alg: No test for adler32 (adler32-zlib) [ 618.545933] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 618.757474] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 620.407526] Key type lgssc registered [ 621.425773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.190525] loop: module loaded [ 628.275680] loop0: detected capacity change from 0 to 204800000 [ 628.482265] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.493571] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.852958] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.342867] loop0: detected capacity change from 0 to 204800000 [ 634.552775] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.574610] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.586543] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.589266] loop0: detected capacity change from 0 to 204800000 [ 647.169900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 647.183447] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.513121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 648.601306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 648.760497] Lustre: lustre-MDT0000: new disk, initializing [ 648.917203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.946632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.632131] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 663.851868] loop1: detected capacity change from 0 to 204800000 [ 669.593244] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.803853] Lustre: lustre-OST07c6: new disk, initializing [ 669.807960] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 669.812177] Lustre: Skipped 1 previous similar message [ 669.818397] Lustre: 26177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 669.888848] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 675.321638] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 679.998579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 680.014212] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 680.044259] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 685.908753] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 685.923045] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 685.933565] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 685.948650] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 685.958750] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 685.968852] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 704.433475] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 704.447640] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.458524] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.469311] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 704.479110] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.489533] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.108383] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 709.118371] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 709.122689] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 709.133113] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.142416] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.152277] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.157866] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 709.166816] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.179027] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 709.192571] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.207453] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.222155] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.421348] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 712.431519] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 712.445676] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.458175] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.466906] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.475979] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.481945] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 712.502289] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.516054] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 712.528189] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.543613] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.556912] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.287508] Lustre: 26525:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 717.297047] Lustre: 26525:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 717.309881] Lustre: 26525:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.319350] Lustre: 26525:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.327314] Lustre: 26525:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.336377] Lustre: 26525:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.353335] Lustre: 26525:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 717.366100] Lustre: 26525:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.372039] Lustre: 26525:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 717.377239] Lustre: 26525:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.382499] Lustre: 26525:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.388335] Lustre: 26525:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.607460] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 728.619437] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 728.633285] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 728.643938] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.649749] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.656248] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.662668] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 728.670980] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.683200] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 728.697316] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.705105] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.721931] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 782.304123] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 782.318232] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.336561] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 785.526164] Lustre: server umount lustre-OST07c6 complete [ 796.153155] Lustre: server umount lustre-MDT0000 complete [ 813.211754] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 816.664189] Key type lgssc unregistered [ 816.994781] LNet: 28737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.007206] LNetError: 28737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.029821] LNet: Removed LNI 192.168.206.138@tcp [ 817.993376] Key type .llcrypt unregistered [ 817.996747] Key type ._llcrypt unregistered [ 838.580824] Key type ._llcrypt registered [ 838.583500] Key type .llcrypt registered [ 838.690669] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 839.898314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.924536] alg: No test for adler32 (adler32-zlib) [ 841.036453] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 841.320214] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 843.063171] Key type lgssc registered [ 844.297728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.998867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 852.050893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.545217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.122687] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 863.719979] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 863.741504] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 866.753933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.221288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 871.684893] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 879.745848] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 885.825285] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 892.900922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 893.318472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.438986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 899.753960] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 907.882929] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.346194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.392631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.396762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.432487] Lustre: Skipped 1 previous similar message [ 935.946557] Lustre: server umount lustre-OST0000 complete [ 943.585411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.587764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.608994] Lustre: Skipped 2 previous similar messages [ 946.439104] Lustre: server umount lustre-MDT0000 complete [ 948.706288] LustreError: 30005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.726709] LustreError: 30005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 951.404054] LustreError: 29985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793100 with bad export cookie 14939974325497224570 [ 951.421657] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.766465] Lustre: server umount lustre-MDT0001 complete [ 963.128603] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 965.831389] Key type lgssc unregistered [ 966.118531] LNet: 33744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.137516] LNetError: 33744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.164140] LNet: Removed LNI 192.168.206.138@tcp [ 967.103414] Key type .llcrypt unregistered [ 967.105187] Key type ._llcrypt unregistered [ 986.756691] Key type ._llcrypt registered [ 986.758979] Key type .llcrypt registered [ 986.834092] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 987.889066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.918609] alg: No test for adler32 (adler32-zlib) [ 989.010352] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 989.283598] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 990.951195] Key type lgssc registered [ 992.090182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.449462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1000.470810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.939029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.402321] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1012.199564] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.241277] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1016.766028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.212432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1022.266796] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1031.303467] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1037.132234] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1044.137858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.550282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.623179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1050.565716] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1059.256227] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.731290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.735727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.766469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.774148] Lustre: Skipped 1 previous similar message [ 1080.455422] Lustre: server umount lustre-OST0000 complete [ 1087.968939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.977738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1088.003377] Lustre: Skipped 2 previous similar messages [ 1088.019284] Lustre: Skipped 1 previous similar message [ 1090.601680] Lustre: server umount lustre-MDT0000 complete [ 1093.095799] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.140799] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1094.405090] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793243 with bad export cookie 17235437630801511100 [ 1094.431166] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.772928] Lustre: server umount lustre-MDT0001 complete [ 1107.247683] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1110.109858] Key type lgssc unregistered [ 1110.394891] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.402886] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.416635] LNet: Removed LNI 192.168.206.138@tcp [ 1111.110429] Key type .llcrypt unregistered [ 1111.112835] Key type ._llcrypt unregistered [ 1138.511485] Key type ._llcrypt registered [ 1138.515468] Key type .llcrypt registered [ 1138.729264] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1140.480220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1140.543104] alg: No test for adler32 (adler32-zlib) [ 1142.084385] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1142.494041] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1144.335172] Key type lgssc registered [ 1145.546201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1153.367383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.388327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.799885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.503723] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1164.263657] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.283662] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1168.357682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.702583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1173.364856] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1181.679211] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.810134] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1194.415345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.848862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.968296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1201.466602] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1209.693409] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.531259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.538224] Lustre: Skipped 1 previous similar message [ 1218.539255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.548909] Lustre: Skipped 1 previous similar message [ 1219.690682] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1223.652204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.665433] Lustre: Skipped 2 previous similar messages [ 1224.762765] Lustre: server umount lustre-MDT0000 complete [ 1224.791795] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.938052] LustreError: 39960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793377 with bad export cookie 16062075202171828745 [ 1227.959439] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.770303] LustreError: 42279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.779413] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.821979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1232.800019] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.812357] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1234.440121] Lustre: server umount lustre-MDT0001 complete [ 1255.570373] Lustre: server umount lustre-OST0000 complete [ 1272.541832] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1275.509559] Key type lgssc unregistered [ 1275.855583] LNet: 43841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.864720] LNetError: 43841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.875872] LNet: Removed LNI 192.168.206.138@tcp [ 1276.677238] Key type .llcrypt unregistered [ 1276.678695] Key type ._llcrypt unregistered [ 1299.537500] Key type ._llcrypt registered [ 1299.540147] Key type .llcrypt registered [ 1299.645467] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1300.685850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.872621] alg: No test for adler32 (adler32-zlib) [ 1301.936546] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1302.193238] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1303.873177] Key type lgssc registered [ 1304.744354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.204956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1312.229430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.608382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.492312] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1322.977848] LustreError: 45256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.012210] LustreError: 45256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1325.821920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.202777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1330.494564] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1338.460356] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1344.872217] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1351.667559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1351.990551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.302497] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1359.166383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1365.494907] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1374.689080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.699314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.734077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.742083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1376.748874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.754199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.345569] Lustre: server umount lustre-OST0000 complete [ 1397.736312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1397.740936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.757575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.762707] Lustre: Skipped 2 previous similar messages [ 1402.336749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.338286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.354742] Lustre: Skipped 1 previous similar message [ 1404.115404] Lustre: server umount lustre-MDT0000 complete [ 1407.086961] LustreError: 45237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793556 with bad export cookie 1920680945236212494 [ 1407.099135] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1407.105162] LustreError: 45237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1413.580977] Lustre: server umount lustre-MDT0001 complete [ 1425.701780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1428.407533] Key type lgssc unregistered [ 1428.694787] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.701657] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.721728] LNet: Removed LNI 192.168.206.138@tcp [ 1429.549808] Key type .llcrypt unregistered [ 1429.552630] Key type ._llcrypt unregistered [ 1456.458188] Key type ._llcrypt registered [ 1456.462362] Key type .llcrypt registered [ 1456.631332] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1457.810269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.894640] alg: No test for adler32 (adler32-zlib) [ 1458.986369] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1459.196638] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1460.831415] Key type lgssc registered [ 1461.907785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.016916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.039237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.468777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.771097] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1480.673461] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.693430] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1483.500988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.013827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.696252] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1496.725630] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.237534] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.904586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.301575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.821033] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1515.457266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1521.591467] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1527.330988] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1530.110111] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1542.253856] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2621h21m28s [ 1543.944199] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1546.212150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.232921] Lustre: Skipped 1 previous similar message [ 1546.233216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.250674] Lustre: Skipped 2 previous similar messages [ 1547.302614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.307111] Lustre: Skipped 2 previous similar messages [ 1549.967787] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1552.204048] Lustre: server umount lustre-MDT0000 complete [ 1553.382676] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.414942] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1555.497994] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1556.108338] LustreError: 52819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.528173] LustreError: 51389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.548980] LustreError: 51389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1561.857033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.023823] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1562.183916] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (not set up) [ 1562.190591] Lustre: Skipped 1 previous similar message [ 1562.342793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.419492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1563.418061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1566.326325] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1569.384421] Lustre: DEBUG MARKER: Wait for df (36642) ... [ 1570.787288] Lustre: DEBUG MARKER: done [ 1576.929108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.938791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.945959] Lustre: Skipped 1 previous similar message [ 1576.949976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.955343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.981847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.630256] Lustre: server umount lustre-OST0000 complete [ 1589.217967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1589.222165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.228296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.228305] Lustre: Skipped 3 previous similar messages [ 1589.264139] Lustre: Skipped 1 previous similar message [ 1590.610945] Lustre: server umount lustre-MDT0000 complete [ 1594.638697] LustreError: 53004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793743 with bad export cookie 18371113757506746748 [ 1594.640468] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.650869] LustreError: 53004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1595.126627] Lustre: server umount lustre-MDT0001 complete [ 1605.139981] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1607.953270] Key type lgssc unregistered [ 1608.215904] LNet: 55299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.224796] LNetError: 55299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.245824] LNet: Removed LNI 192.168.206.138@tcp [ 1609.030319] Key type .llcrypt unregistered [ 1609.032176] Key type ._llcrypt unregistered [ 1648.623739] Key type ._llcrypt registered [ 1648.628260] Key type .llcrypt registered [ 1648.776669] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1649.960774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.989264] alg: No test for adler32 (adler32-zlib) [ 1651.024688] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1651.237298] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1652.880757] Key type lgssc registered [ 1653.855772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.296735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.317082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.577206] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1662.601470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1662.742739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.794859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1674.109462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.173426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1674.199369] Lustre: 57504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1674.478623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1678.834300] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1685.331306] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1690.677221] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1697.328241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1697.617158] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1697.797679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.760834] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1704.620983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1711.414634] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.698538] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2621h24m26s [ 1720.185747] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1727.322691] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1729.732945] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1729.794189] LustreError: 57512:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1750.156522] Lustre: lustre-MDT0000: Client a16172f3-8aba-4abe-9279-c00e6da45586 (at 192.168.206.38@tcp) reconnecting [ 1769.800886] LustreError: 57512:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1772.224258] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1774.618780] Lustre: DEBUG MARKER: done [ 1781.221172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.229919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.252628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.729540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.738571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.762303] Lustre: server umount lustre-OST0000 complete [ 1786.848169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.861416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.868506] Lustre: Skipped 2 previous similar messages [ 1787.940761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.948708] Lustre: Skipped 1 previous similar message [ 1793.105350] Lustre: server umount lustre-MDT0000 complete [ 1796.725037] LustreError: 57495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793945 with bad export cookie 15181882895797319554 [ 1796.727670] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.734647] LustreError: 57495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1796.994568] Lustre: server umount lustre-MDT0001 complete [ 1808.595371] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1811.465269] Key type lgssc unregistered [ 1811.768042] LNet: 61831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.779587] LNetError: 61831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.843773] LNet: Removed LNI 192.168.206.138@tcp [ 1813.887709] Key type .llcrypt unregistered [ 1813.889792] Key type ._llcrypt unregistered [ 1865.057861] Key type ._llcrypt registered [ 1865.065575] Key type .llcrypt registered [ 1865.146420] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1866.540392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.631805] alg: No test for adler32 (adler32-zlib) [ 1867.953352] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1868.532711] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1870.319208] Key type lgssc registered [ 1871.448570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1875.419840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1875.448341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.847197] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1876.870605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1876.974263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.872083] Lustre: Failing over lustre-MDT0000 [ 1878.235563] Lustre: server umount lustre-MDT0000 complete [ 1885.923535] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:27:12 (1787794032) [ 1894.377931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.809894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.862850] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1909.879476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.059148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1910.138345] Lustre: 64629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1910.478829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.432149] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1923.856575] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.113480] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.274148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.501888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1937.684373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.839579] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1943.146208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1950.209574] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1957.312417] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1967.075081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1967.083110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.098521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.617437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.618035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.638650] Lustre: server umount lustre-MDT0000 complete [ 1975.374136] LustreError: 64619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794124 with bad export cookie 5782380764306616947 [ 1975.384606] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.394345] LustreError: 64619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.768913] Lustre: server umount lustre-MDT0001 complete [ 1980.199227] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1985.517494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1993.969594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.231673] Lustre: 63548:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794128/real 1787794128] req@000000008119fc36 x1874637899211776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.255536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.265216] Lustre: Skipped 1 previous similar message [ 1997.343155] Lustre: 63548:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794130/real 1787794130] req@000000009cbaa3cd x1874637899211904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.351122] Lustre: 63550:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794133/real 1787794133] req@00000000637e123e x1874637899212032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.415844] LustreError: 63546:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f6f3e94b x1874637899212928/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.734217] LustreError: 66487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.758627] LustreError: 66487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2000.835774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.930189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2004.837065] LustreError: 66488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.126419] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2005.859508] LustreError: 66487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.860901] Lustre: 63548:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794139/real 1787794139] req@000000000055eb9f x1874637899212416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2005.880637] LustreError: 66487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2005.911382] Lustre: 63548:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2011.108022] Lustre: 63547:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794144/real 1787794144] req@000000007de396db x1874637899212544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.111111] LustreError: 68411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.158368] LustreError: 68411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2012.675157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.027278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.261804] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2018.284470] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.447745] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.809076] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.762442] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.752687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2038.760187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.774450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.786103] Lustre: Skipped 2 previous similar messages [ 2043.751793] Lustre: server umount lustre-MDT0000 complete [ 2044.913126] LustreError: 68411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.935606] LustreError: 68411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2047.493490] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794196 with bad export cookie 5782380764306619222 [ 2047.502597] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.507903] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.894503] Lustre: server umount lustre-MDT0001 complete [ 2053.781936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2062.987790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.400041] Lustre: 63549:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794199/real 1787794199] req@0000000084d7fb0a x1874637899239936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.432720] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.447490] Lustre: Skipped 2 previous similar messages [ 2073.098533] LustreError: 66488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.211470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.309164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2077.766705] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2086.269290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.657932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2092.011514] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2095.088271] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2099.718060] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.995708] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.556161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.559952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.573345] Lustre: Skipped 1 previous similar message [ 2115.597863] Lustre: Skipped 5 previous similar messages [ 2120.674020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.680073] Lustre: Skipped 2 previous similar messages [ 2127.327532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2127.506282] Lustre: server umount lustre-MDT0000 complete [ 2130.915086] LustreError: 71093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.940312] LustreError: 71093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2131.070769] LustreError: 65364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794280 with bad export cookie 5782380764306620601 [ 2131.080825] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.084769] LustreError: 65364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2131.386842] Lustre: server umount lustre-MDT0001 complete [ 2137.625443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2141.183588] Lustre: DEBUG MARKER: files compared the same [ 2153.326226] Lustre: server umount lustre-OST0000 complete [ 2168.341984] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2170.722396] Key type lgssc unregistered [ 2170.957555] LNet: 74368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.966855] LNetError: 74368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.983961] LNet: Removed LNI 192.168.206.138@tcp [ 2171.762183] Key type .llcrypt unregistered [ 2171.764755] Key type ._llcrypt unregistered [ 2196.788316] Key type ._llcrypt registered [ 2196.790205] Key type .llcrypt registered [ 2196.905368] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2197.785256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2197.930148] alg: No test for adler32 (adler32-zlib) [ 2199.130830] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2199.382296] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2201.055452] Key type lgssc registered [ 2202.117570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.099599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2210.127418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.784844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.566538] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2221.031658] LustreError: 75785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.051399] LustreError: 75785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2224.955045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.199058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.303773] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2236.749991] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.297653] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2248.969493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.440651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.258386] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2256.627362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2262.743451] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2271.201400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.210220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.224624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.224025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.235988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.255857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2273.044479] Lustre: server umount lustre-OST0000 complete [ 2277.345546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.360236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.364457] Lustre: Skipped 2 previous similar messages [ 2282.464161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.473162] Lustre: Skipped 1 previous similar message [ 2282.744419] Lustre: server umount lustre-MDT0000 complete [ 2286.142616] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794435 with bad export cookie 4816300165675976178 [ 2286.147416] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.155926] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.558786] Lustre: server umount lustre-MDT0001 complete [ 2298.280361] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2301.018228] Key type lgssc unregistered [ 2301.251339] LNet: 79315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.259158] LNetError: 79315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.314589] LNet: Removed LNI 192.168.206.138@tcp [ 2303.073422] Key type .llcrypt unregistered [ 2303.077844] Key type ._llcrypt unregistered [ 2326.591333] Key type ._llcrypt registered [ 2326.594311] Key type .llcrypt registered [ 2326.758824] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2328.192218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.229867] alg: No test for adler32 (adler32-zlib) [ 2329.300974] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2329.597099] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2331.319183] Key type lgssc registered [ 2332.397940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2340.337955] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2340.362952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.065175] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2361.183330] LustreError: 80701:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 2387.874994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.081958] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2403.166624] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.346667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.431335] Lustre: *** cfs_fail_loc=706, val=0*** [ 2410.788596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.864327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2415.009237] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2416.108368] LustreError: 80727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.125606] LustreError: 80727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2421.216307] LustreError: 80726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.251520] LustreError: 81785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.272515] LustreError: 81785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2422.688118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.974373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.631798] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2432.178657] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2436.427040] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2438.629184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.630979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2438.632948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2438.644769] Lustre: Skipped 1 previous similar message [ 2438.653411] Lustre: Skipped 1 previous similar message [ 2443.746334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2443.762769] Lustre: Skipped 1 previous similar message [ 2444.381949] Lustre: server umount lustre-OST0000 complete [ 2448.864122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.872043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.879934] Lustre: Skipped 1 previous similar message [ 2448.888886] Lustre: Skipped 1 previous similar message [ 2453.984847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2453.990713] Lustre: Skipped 1 previous similar message [ 2459.105167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.117140] Lustre: Skipped 1 previous similar message [ 2462.175460] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.348559] Lustre: server umount lustre-MDT0000 complete [ 2464.228246] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.248866] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2465.665823] LustreError: 80707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794614 with bad export cookie 13474245327988495614 [ 2465.670563] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.675805] LustreError: 80707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2465.939933] Lustre: server umount lustre-MDT0001 complete [ 2475.902087] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2478.129871] Key type lgssc unregistered [ 2478.396239] LNet: 84315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.412291] LNetError: 84315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.427896] LNet: Removed LNI 192.168.206.138@tcp [ 2479.107622] Key type .llcrypt unregistered [ 2479.110698] Key type ._llcrypt unregistered [ 2500.634933] Key type ._llcrypt registered [ 2500.637551] Key type .llcrypt registered [ 2500.706345] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2501.449799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2501.476740] alg: No test for adler32 (adler32-zlib) [ 2502.483755] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2502.657253] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2504.327879] Key type lgssc registered [ 2505.184295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2510.842246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2510.873411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.013550] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2522.710051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.240050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2526.830612] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2528.737932] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.768976] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2533.859503] LustreError: 86417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.878052] LustreError: 86417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2534.319797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.679168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.003704] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.037257] LustreError: 86414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2542.058257] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2545.120517] LustreError: 86418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.136096] LustreError: 86418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2549.217310] LustreError: 86417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.243539] LustreError: 86417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2549.732622] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.742068] Lustre: MGS: Client 9d7ad0fe-bb8d-4800-ac26-1050f762dd09 (at 0@lo) reconnecting [ 2549.762393] LustreError: 86428:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2549.917992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.009078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2554.659253] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2570.724089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.738094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.747080] LustreError: Skipped 1 previous similar message [ 2570.748180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2570.764754] Lustre: Skipped 1 previous similar message [ 2575.174946] Lustre: server umount lustre-OST0000 complete [ 2580.963398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2580.969183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.979160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.984876] Lustre: Skipped 2 previous similar messages [ 2584.673914] Lustre: server umount lustre-MDT0000 complete [ 2586.080612] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.098571] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.098718] Lustre: server umount lustre-MDT0001 complete [ 2592.711423] Lustre: server umount MGS complete [ 2602.874273] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:39:09 (1787794749) [ 2626.484232] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2635.389942] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2664.901800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.503269] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2676.900607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.084809] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2677.124819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2677.185676] Lustre: lustre-MDT0001: new disk, initializing [ 2677.249077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.264984] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2677.276171] Lustre: 93595:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2677.288129] LustreError: 93595:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2680.971859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2690.005802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.283731] Lustre: lustre-OST0000: new disk, initializing [ 2690.286633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2690.299862] Lustre: 94362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2690.365166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.997937] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2696.676090] Lustre: 94368:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000067a981a1 x1874638564006912/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787794861 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2704.538093] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2710.266311] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.283068] Lustre: MGS: Client c076813d-a703-45e1-9475-77888c605429 (at 0@lo) reconnecting [ 2710.314451] Lustre: 92865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2710.326721] LustreError: 93581:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2710.352483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2710.423936] Lustre: lustre-MDT0000: new disk, initializing [ 2710.566508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.594332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2710.720768] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2711.649799] Lustre: 95431:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000017630d5c x1874638564013696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787794876 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2714.579485] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2717.157208] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2720.751524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2720.765708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2720.900167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2743.779562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.783062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.804740] Lustre: Skipped 2 previous similar messages [ 2743.816509] Lustre: Skipped 1 previous similar message [ 2748.905646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.914694] Lustre: Skipped 1 previous similar message [ 2749.731613] Lustre: server umount lustre-OST0000 complete [ 2754.019763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2754.021223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.027709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.037566] Lustre: Skipped 1 previous similar message [ 2764.260756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.272358] Lustre: Skipped 4 previous similar messages [ 2767.328084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2767.409633] Lustre: server umount lustre-MDT0000 complete [ 2769.377597] LustreError: 93567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.386279] LustreError: 93567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2785.247159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2785.486075] Lustre: server umount lustre-MDT0001 complete [ 2788.995231] Lustre: server umount MGS complete [ 2799.010134] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:42:26 (1787794946) [ 2815.145245] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2817.558966] Key type lgssc unregistered [ 2817.757811] LNet: 97930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.770560] LNetError: 97930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.786597] LNet: Removed LNI 192.168.206.138@tcp [ 2818.411488] Key type .llcrypt unregistered [ 2818.413322] Key type ._llcrypt unregistered [ 2834.212939] Key type ._llcrypt registered [ 2834.215834] Key type .llcrypt registered [ 2834.300076] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2835.351899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2835.409274] alg: No test for adler32 (adler32-zlib) [ 2836.568531] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2836.826734] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2838.567511] Key type lgssc registered [ 2839.672322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2846.323594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2846.338335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.662974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.237567] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2856.933975] LustreError: 99150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.951399] LustreError: 99150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2859.185711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.492212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.348163] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2870.270523] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2875.637970] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2882.155745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2882.528204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.349065] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2892.805719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2896.941145] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2905.057026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.068806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.099448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.133987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.137955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.139646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.168141] Lustre: Skipped 1 previous similar message [ 2910.674591] Lustre: server umount lustre-OST0000 complete [ 2915.811080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2915.824230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.844896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.371505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.375473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.855394] Lustre: server umount lustre-MDT0000 complete [ 2922.515056] LustreError: 99130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795071 with bad export cookie 5551980173175251032 [ 2922.519779] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.534146] LustreError: 99130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2922.818349] Lustre: server umount lustre-MDT0001 complete [ 2932.035899] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2934.584723] Key type lgssc unregistered [ 2934.852261] LNet: 102681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.867800] LNetError: 102681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.893480] LNet: Removed LNI 192.168.206.138@tcp [ 2935.637289] Key type .llcrypt unregistered [ 2935.639900] Key type ._llcrypt unregistered [ 2949.803233] Key type ._llcrypt registered [ 2949.808625] Key type .llcrypt registered [ 2949.915520] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2950.702571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2950.835512] alg: No test for adler32 (adler32-zlib) [ 2951.862248] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2952.150647] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2953.879254] Key type lgssc registered [ 2954.849704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2960.129406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2960.132149] LustreError: 103758:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2960.151058] Lustre: Skipped 1 previous similar message [ 2960.151079] LustreError: 103759:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2960.171963] LustreError: 103758:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2960.177214] LustreError: 103758:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2960.182176] LustreError: 103758:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2960.186777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.631829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.997228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.200542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.424430] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2978.373189] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lsmod [ 2981.689540] LustreError: 104966:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2981.702478] LustreError: 104965:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2981.710928] LustreError: 104966:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 2981.720719] LustreError: 104966:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2981.737347] LustreError: 104966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2981.749172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.309833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.489189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.492254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.509524] Lustre: Skipped 2 previous similar messages [ 2987.517406] Lustre: Skipped 2 previous similar messages [ 2991.416413] Lustre: server umount lustre-MDT0000 complete [ 2992.609900] LustreError: 103791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.632954] LustreError: 103791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2994.657958] LustreError: 104230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.696485] LustreError: 104230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2994.893650] LustreError: 103774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795144 with bad export cookie 13937807846629149366 [ 2994.914790] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.831365] LustreError: 104227:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006bcd14b9 x1874639035846912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2997.029449] Lustre: server umount lustre-MDT0001 complete [ 3011.011902] Lustre: server umount lustre-OST0000 complete [ 3018.072798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.494712] LustreError: 106029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.566737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.066137] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3023.846912] LustreError: 106030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.937795] LustreError: 106029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.691826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.015340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3034.768112] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3041.705835] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3047.434871] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3054.157887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.547186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.378870] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3063.819568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3067.253874] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3077.089206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.111133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.128953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3079.143652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3079.144698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3079.166750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.364704] Lustre: server umount lustre-OST0000 complete [ 3084.266199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.280088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.286500] Lustre: Skipped 1 previous similar message [ 3084.306638] Lustre: Skipped 2 previous similar messages [ 3089.377414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.391047] Lustre: Skipped 1 previous similar message [ 3090.281258] Lustre: server umount lustre-MDT0000 complete [ 3093.921718] LustreError: 108810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795243 with bad export cookie 13937807846629150080 [ 3093.930209] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.931173] LustreError: 108810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3094.233721] Lustre: server umount lustre-MDT0001 complete [ 3105.200492] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3107.648827] Key type lgssc unregistered [ 3107.936851] LNet: 109561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3107.944982] LNetError: 109561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3107.954960] LNet: Removed LNI 192.168.206.138@tcp [ 3108.912149] Key type .llcrypt unregistered [ 3108.925429] Key type ._llcrypt unregistered [ 3131.974190] Key type ._llcrypt registered [ 3131.978434] Key type .llcrypt registered [ 3132.111691] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3133.714350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3133.818278] alg: No test for adler32 (adler32-zlib) [ 3134.799793] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 3134.980236] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3136.623176] Key type lgssc registered [ 3137.586419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3145.179641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3145.201879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.751065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.283361] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3155.942558] LustreError: 110975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.963030] LustreError: 110975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3159.768276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.136229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.537139] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3171.619741] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3176.256673] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3181.554725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3181.872945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.404263] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3191.090782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3191.945702] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3197.082792] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3199.387763] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3205.600961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.614680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.625421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.624648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.627970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.628435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3207.763489] Lustre: server umount lustre-OST0000 complete [ 3211.232862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3211.237466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.244975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3211.248316] Lustre: Skipped 1 previous similar message [ 3216.782416] Lustre: server umount lustre-MDT0000 complete [ 3216.867819] LustreError: 110976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.882206] LustreError: 110976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3219.761707] LustreError: 110957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795368 with bad export cookie 597530240982315165 [ 3219.763555] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.770691] LustreError: 110957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.034141] Lustre: server umount lustre-MDT0001 complete [ 3228.429607] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3230.849095] Key type lgssc unregistered [ 3231.159248] LNet: 114609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.173557] LNetError: 114609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3232.234075] LNet: Removed LNI 192.168.206.138@tcp [ 3233.097325] Key type .llcrypt unregistered [ 3233.099273] Key type ._llcrypt unregistered [ 3248.775174] Key type ._llcrypt registered [ 3248.777545] Key type .llcrypt registered [ 3248.871856] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3249.805025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3249.979792] alg: No test for adler32 (adler32-zlib) [ 3250.997959] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 3251.169575] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3252.783687] Key type lgssc registered [ 3253.732717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3260.213700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3260.224423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.565554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.220631] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3271.139357] LustreError: 115829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.150141] LustreError: 115829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3271.946370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.206022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3275.727588] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3281.737886] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3286.490616] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3292.908978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.286967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.059703] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3306.384272] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3306.605722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3317.218108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.222349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.227911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.258165] Lustre: Skipped 1 previous similar message [ 3319.623504] Lustre: server umount lustre-OST0000 complete [ 3323.360731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3323.380425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.399123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3323.407090] Lustre: Skipped 1 previous similar message [ 3327.459607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.461146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.485820] Lustre: Skipped 1 previous similar message [ 3332.576953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.595095] Lustre: Skipped 1 previous similar message [ 3337.183401] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3337.432303] Lustre: server umount lustre-MDT0000 complete [ 3337.700099] LustreError: 118188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.723313] LustreError: 118188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3341.484218] LustreError: 118322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795490 with bad export cookie 14202789584426468703 [ 3341.492958] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.497267] LustreError: 118322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3342.039804] Lustre: server umount lustre-MDT0001 complete [ 3352.451085] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3355.125333] Key type lgssc unregistered [ 3355.402063] LNet: 119362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3355.419906] LNetError: 119362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3356.455329] LNet: Removed LNI 192.168.206.138@tcp [ 3357.262524] Key type .llcrypt unregistered [ 3357.264338] Key type ._llcrypt unregistered [ 3380.344411] Key type ._llcrypt registered [ 3380.346858] Key type .llcrypt registered [ 3380.450154] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3381.174790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3381.218030] alg: No test for adler32 (adler32-zlib) [ 3382.389218] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 3382.592462] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3384.239128] Key type lgssc registered [ 3385.472379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3392.622562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3392.649348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.197604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.062354] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3404.770826] LustreError: 120778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.794539] LustreError: 120778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3409.288837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.760804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3414.521366] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3422.459476] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3429.238428] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3436.063424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.380653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.483602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3443.332904] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3450.432547] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3460.721745] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3464.102278] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3464.112768] Lustre: Skipped 2 previous similar messages [ 3468.149797] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3468.158543] Lustre: Skipped 2 previous similar messages [ 3469.384840] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3469.418316] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3476.256917] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3476.264114] Lustre: Skipped 2 previous similar messages [ 3477.540147] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp [ 3483.501651] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.38/24@tcp in log lustre-MDT0000 [ 3483.511164] Lustre: Skipped 2 previous similar messages [ 3485.520573] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 3485.527922] Lustre: Skipped 1 previous similar message [ 3495.905902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.909528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3495.913968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.927597] Lustre: Skipped 1 previous similar message [ 3500.698566] Lustre: server umount lustre-OST0000 complete [ 3506.146367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.161014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.168653] Lustre: Skipped 2 previous similar messages [ 3510.335572] Lustre: server umount lustre-MDT0000 complete [ 3511.274232] LustreError: 123045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.296035] LustreError: 123045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3513.766947] LustreError: 120760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795662 with bad export cookie 18267304601782674792 [ 3513.768389] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3513.775659] LustreError: 120760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3514.116987] Lustre: server umount lustre-MDT0001 complete [ 3523.691738] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3526.335753] Key type lgssc unregistered [ 3526.605204] LNet: 125468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3526.613776] LNetError: 125468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3526.632307] LNet: Removed LNI 192.168.206.138@tcp [ 3527.559948] Key type .llcrypt unregistered [ 3527.566736] Key type ._llcrypt unregistered [ 3553.959432] Key type ._llcrypt registered [ 3553.961935] Key type .llcrypt registered [ 3554.082703] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3555.224157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3555.271472] alg: No test for adler32 (adler32-zlib) [ 3556.293989] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 3556.485902] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3558.135228] Key type lgssc registered [ 3559.248518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3566.751270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3566.784669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.100497] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3568.129566] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.38/24@tcp [ 3568.135942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.113625] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3578.342053] LustreError: 127029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.356282] LustreError: 127029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3580.094836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.380459] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3580.413903] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 3580.465519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3584.606634] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3591.660252] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3596.841411] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3602.660403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3602.982607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.088924] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3613.245807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3616.315649] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3626.996031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3627.008241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.025770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.515540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.518805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.552514] Lustre: Skipped 1 previous similar message [ 3633.169620] Lustre: server umount lustre-OST0000 complete [ 3636.705602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3636.714815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.732979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.766546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.773254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.576502] Lustre: server umount lustre-MDT0000 complete [ 3643.872311] LustreError: 128388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.904689] LustreError: 127008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795795 with bad export cookie 465547437331023989 [ 3645.909553] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.919507] LustreError: 127008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3646.237565] Lustre: server umount lustre-MDT0001 complete [ 3656.924177] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3659.084927] Key type lgssc unregistered [ 3659.362650] LNet: 130615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3659.369978] LNetError: 130615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3659.393095] LNet: Removed LNI 192.168.206.138@tcp [ 3660.187872] Key type .llcrypt unregistered [ 3660.189736] Key type ._llcrypt unregistered [ 3684.557992] Key type ._llcrypt registered [ 3684.560039] Key type .llcrypt registered [ 3684.673647] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3698.188460] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3699.215182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.230796] alg: No test for adler32 (adler32-zlib) [ 3700.358458] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 3700.620698] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3702.311259] Key type lgssc registered [ 3703.496479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3735.933864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3735.954250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.139557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3737.182805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3737.270990] Lustre: lustre-MDT0000: new disk, initializing [ 3737.345903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.373059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3741.026589] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3749.921638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.031673] Lustre: 134012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3750.103750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3750.107420] Lustre: Skipped 1 previous similar message [ 3750.196671] Lustre: lustre-MDT0001: new disk, initializing [ 3750.279885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.327146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3750.337803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3754.090750] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3762.501375] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3767.459830] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3774.132142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3774.334397] Lustre: lustre-OST0000: new disk, initializing [ 3774.336783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3774.345428] Lustre: 135977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3774.407555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.101675] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3782.217645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3782.227823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3782.316551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.313595] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.958265] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3794.161629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3798.200404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3798.378278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3803.105015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.116139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.127420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.137084] Lustre: Skipped 1 previous similar message [ 3803.154593] Lustre: Skipped 1 previous similar message [ 3808.247208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.264803] Lustre: Skipped 1 previous similar message [ 3813.357421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.363388] Lustre: Skipped 1 previous similar message [ 3814.368379] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3814.580751] Lustre: server umount lustre-OST0000 complete [ 3818.468633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.477198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.488976] Lustre: Skipped 1 previous similar message [ 3818.501917] Lustre: Skipped 1 previous similar message [ 3823.586315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.600222] Lustre: Skipped 1 previous similar message [ 3823.978221] Lustre: server umount lustre-MDT0000 complete [ 3828.023686] LustreError: 134005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795977 with bad export cookie 3454339557146986086 [ 3828.032878] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.037138] LustreError: 134005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3828.833549] Lustre: server umount lustre-MDT0001 complete [ 3837.853696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.693692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.214973] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3844.065233] LustreError: 138197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.094304] LustreError: 138197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3848.166582] LustreError: 138198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.999971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.281497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3853.574668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.439594] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3866.803400] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3872.373798] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3878.408890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.706740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.407074] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3891.702315] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3897.290539] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3897.477418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3901.998547] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3902.229538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.116922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.304429] Lustre: lustre-OST0001: new disk, initializing [ 3911.308391] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3911.317159] Lustre: 141570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3911.402055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3917.381362] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3919.929572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3919.939643] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3920.038983] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3925.084494] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3925.355544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3929.372442] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3929.620867] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3934.397501] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956c4458a000.ost_server_uuid 50 [ 3935.728860] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956c4458a000.ost_server_uuid in IDLE state after 0 sec [ 3945.959922] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3945.963132] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.965921] LustreError: Skipped 2 previous similar messages [ 3945.968808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3945.976223] Lustre: Skipped 1 previous similar message [ 3945.991187] Lustre: Skipped 1 previous similar message [ 3948.724294] Lustre: server umount lustre-OST0001 complete [ 3951.079766] LustreError: 140021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.095078] LustreError: 140021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3954.660832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.666766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.198047] LustreError: 140018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.208868] LustreError: 140018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3958.046646] Lustre: server umount lustre-OST0000 complete [ 3961.312805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.317597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.326102] LustreError: Skipped 1 previous similar message [ 3961.349123] Lustre: Skipped 1 previous similar message [ 3966.972815] Lustre: server umount lustre-MDT0000 complete [ 3969.794786] LustreError: 138177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796118 with bad export cookie 3454339557146986961 [ 3969.798733] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3969.806612] LustreError: 138177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3970.091458] Lustre: server umount lustre-MDT0001 complete [ 3987.656470] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3989.908493] Key type lgssc unregistered [ 3990.099682] LNet: 144585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3990.108683] LNetError: 144585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.119878] LNet: Removed LNI 192.168.206.138@tcp [ 3990.818613] Key type .llcrypt unregistered [ 3990.823318] Key type ._llcrypt unregistered [ 4033.772443] Key type ._llcrypt registered [ 4033.776139] Key type .llcrypt registered [ 4033.874098] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4048.976556] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4049.659797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4049.890089] alg: No test for adler32 (adler32-zlib) [ 4050.928345] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 4051.150912] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4052.799161] Key type lgssc registered [ 4053.861625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4088.149087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4088.183652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.518608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4089.585025] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4089.685186] Lustre: lustre-MDT0000: new disk, initializing [ 4089.766974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.778818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4094.166131] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4103.775834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.877808] Lustre: 148735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4103.950094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4103.955608] Lustre: Skipped 1 previous similar message [ 4104.023907] Lustre: lustre-MDT0001: new disk, initializing [ 4104.101643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.142987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4104.160852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4107.938929] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4116.859486] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4122.818519] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4129.495230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.744628] Lustre: lustre-OST0000: new disk, initializing [ 4129.747515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4129.753239] Lustre: 150701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4129.832755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.179220] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4138.085627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4138.095503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4138.198214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4144.536807] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4149.454673] Lustre: Failing over lustre-OST0000 [ 4149.746817] Lustre: server umount lustre-OST0000 complete [ 4167.648246] Lustre: 146958:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796300/real 1787796300] req@0000000068e7a12d x1874640188244224/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787796316 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4167.688326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.545644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.869584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.926452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.942309] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4169.951110] Lustre: 146959:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796302/real 1787796302] req@00000000cf11112b x1874640188244352/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787796318 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4169.979232] Lustre: 146959:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4171.743256] Lustre: 146958:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796303/real 1787796303] req@00000000606c8e0c x1874640188245120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787796319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4172.909684] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4174.943130] Lustre: 146957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796308/real 1787796308] req@0000000044261988 x1874640188246272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787796324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4175.243282] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4177.091711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4177.091753] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4179.043237] Lustre: Failing over lustre-MDT0000 [ 4179.268611] Lustre: server umount lustre-MDT0000 complete [ 4180.063270] Lustre: 146959:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796313/real 1787796313] req@00000000058e65e2 x1874640188247424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787796329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4180.097322] Lustre: 146959:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4180.964075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4180.988187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.039833] LustreError: 149540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.070503] LustreError: 149540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4182.496248] LustreError: 148746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.523974] LustreError: 148746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4184.195695] LustreError: 148741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.623181] LustreError: 151018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.642072] LustreError: 151018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4192.739204] LustreError: 148740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.760144] LustreError: 148740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4197.868299] Lustre: 146960:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796331/real 1787796331] req@000000008016c5b7 x1874640188257920/t0(0) o400->MGC192.168.206.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787796347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4197.905160] Lustre: 146960:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4197.912940] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4199.321776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.978689] LustreError: 148745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.004775] LustreError: 148745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4208.102824] LustreError: 146956:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f21d2e5b x1874640188262784/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4208.352378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.424032] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4209.246221] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4213.054752] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4213.738600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4213.747821] Lustre: Skipped 1 previous similar message [ 4213.764365] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4213.790545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4223.456077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4223.469140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.482955] Lustre: Skipped 2 previous similar messages [ 4223.494404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4228.536647] Lustre: server umount lustre-OST0000 complete [ 4234.209853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.213033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4234.215429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.215436] Lustre: Skipped 1 previous similar message [ 4234.223723] Lustre: Skipped 2 previous similar messages [ 4239.301587] Lustre: server umount lustre-MDT0000 complete [ 4239.329821] LustreError: 148745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.342678] LustreError: 148745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4242.937959] LustreError: 148727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796392 with bad export cookie 2109421763394044068 [ 4242.953618] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.966113] LustreError: 148727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4243.309881] Lustre: server umount lustre-MDT0001 complete [ 4255.017615] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4257.926685] Key type lgssc unregistered [ 4258.236512] LNet: 154359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4258.273471] LNetError: 154359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4258.307781] LNet: Removed LNI 192.168.206.138@tcp [ 4259.309269] Key type .llcrypt unregistered [ 4259.311118] Key type ._llcrypt unregistered [ 4295.900331] Key type ._llcrypt registered [ 4295.904163] Key type .llcrypt registered [ 4296.041434] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4296.958715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4297.091760] alg: No test for adler32 (adler32-zlib) [ 4298.125614] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 4298.384772] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4300.023181] Key type lgssc registered [ 4300.869827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4308.524793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4308.544805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.049538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.631436] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4320.260195] LustreError: 155876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.303386] LustreError: 155876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4323.871726] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.352285] LustreError: 155877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.057574] LustreError: 155905:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4328.076982] LustreError: 155905:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4328.085683] LustreError: 155905:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4328.600612] Lustre: server umount lustre-MDT0000 complete [ 4345.666811] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4355.423801] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4390.858324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.038793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4391.096393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4391.165775] Lustre: lustre-MDT0000: new disk, initializing [ 4391.227636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.246879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4396.042764] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4406.822803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.132489] Lustre: 159210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4407.286721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4407.291194] Lustre: Skipped 1 previous similar message [ 4407.356872] Lustre: lustre-MDT0001: new disk, initializing [ 4407.429697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.471570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4407.480247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4412.356732] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4421.969052] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4428.174121] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4435.286831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.626340] Lustre: lustre-OST0000: new disk, initializing [ 4435.636297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4435.645790] Lustre: 161180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4435.742458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.695167] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4442.724743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4442.753547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4442.857041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4450.769455] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4456.023771] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4457.986526] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:10:05 (1787796605) [ 4714.465343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.466340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4714.478737] Lustre: Skipped 2 previous similar messages [ 4714.488706] Lustre: Skipped 2 previous similar messages [ 4716.485837] Lustre: server umount lustre-MDT0000 complete [ 4719.587199] LustreError: 159222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.605843] LustreError: 159222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4720.154559] LustreError: 159202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796869 with bad export cookie 9363130816956459723 [ 4720.156317] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.162832] LustreError: 159202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4724.705616] LustreError: 159217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.706410] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.736894] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4725.743151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4726.840443] Lustre: server umount lustre-MDT0001 complete [ 4745.183159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4745.425640] Lustre: server umount lustre-OST0000 complete [ 4754.928630] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4765.008235] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4801.787981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.996234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4802.035441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4802.114941] Lustre: lustre-MDT0000: new disk, initializing [ 4802.256807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.274423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4806.493986] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4816.741610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.913436] Lustre: 165845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4817.009390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4817.016898] Lustre: Skipped 1 previous similar message [ 4817.123177] Lustre: lustre-MDT0001: new disk, initializing [ 4817.253111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.307162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4817.321480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4822.087078] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4832.397635] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.698547] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4845.719132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.942901] Lustre: lustre-OST0000: new disk, initializing [ 4845.948883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.954576] Lustre: 167812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4846.030747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.178503] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4852.332588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4852.351274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4852.464671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4861.716979] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4866.712133] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4866.913972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4871.374257] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4871.599351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4878.308939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.316032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.329807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4878.331957] LustreError: Skipped 1 previous similar message [ 4878.386202] Lustre: Skipped 1 previous similar message [ 4879.808404] Lustre: server umount lustre-OST0000 complete [ 4888.544595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.571767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.575417] Lustre: Skipped 1 previous similar message [ 4890.600868] Lustre: server umount lustre-MDT0000 complete [ 4893.667418] LustreError: 165855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.691077] LustreError: 165855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4894.920875] LustreError: 167811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797044 with bad export cookie 9363130816956618553 [ 4894.932390] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.933736] LustreError: 167811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4895.219956] Lustre: server umount lustre-MDT0001 complete [ 4904.264475] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:17:31 (1787797051) [ 4922.062426] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4931.941678] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4970.825124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.059460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4971.086432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4971.149392] Lustre: lustre-MDT0000: new disk, initializing [ 4971.234353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.260851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4976.265256] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4987.723831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.889305] Lustre: 172566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4988.116287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4993.148354] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5002.467464] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5008.799994] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5016.254800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5016.588412] Lustre: lustre-OST0000: new disk, initializing [ 5016.591363] Lustre: Skipped 1 previous similar message [ 5016.594721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5016.600864] Lustre: Skipped 2 previous similar messages [ 5016.605517] Lustre: 174543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5023.178413] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5025.881211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5025.891576] Lustre: Skipped 1 previous similar message [ 5025.903557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5026.052511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5033.535219] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5041.635932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5041.643179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.646025] LustreError: Skipped 1 previous similar message [ 5041.646991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5041.686496] Lustre: Skipped 2 previous similar messages [ 5047.211199] Lustre: server umount lustre-OST0000 complete [ 5051.874431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.895789] Lustre: Skipped 1 previous similar message [ 5057.290404] Lustre: server umount lustre-MDT0000 complete [ 5060.929298] LustreError: 172559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797210 with bad export cookie 9363130816956619435 [ 5060.940248] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.952085] LustreError: 172559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5061.225362] Lustre: server umount lustre-MDT0001 complete [ 5071.160458] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:20:18 (1787797218) [ 5089.180532] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5098.546539] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5138.979588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.413651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5139.459724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5139.644453] Lustre: lustre-MDT0000: new disk, initializing [ 5139.759427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.770231] Lustre: Skipped 2 previous similar messages [ 5139.806342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5145.220816] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5156.668595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.854162] Lustre: 178873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.162784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5162.135669] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5172.317690] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5179.033350] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5185.786821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.017738] Lustre: 180848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5192.326926] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5194.346446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5194.470702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5203.517342] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5215.201508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5215.202924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.217685] LustreError: Skipped 1 previous similar message [ 5215.254280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5215.267702] Lustre: Skipped 8 previous similar messages [ 5217.994889] Lustre: server umount lustre-OST0000 complete [ 5223.915057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5228.369923] Lustre: server umount lustre-MDT0000 complete [ 5230.561596] LustreError: 178879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.582083] LustreError: 178879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5232.754821] LustreError: 180847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797381 with bad export cookie 9363130816956620520 [ 5232.771927] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.774178] LustreError: 180847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5233.323522] Lustre: server umount lustre-MDT0001 complete [ 5245.379708] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5248.117041] Key type lgssc unregistered [ 5248.505134] LNet: 182703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.516676] LNetError: 182703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5248.544778] LNet: Removed LNI 192.168.206.138@tcp [ 5249.497450] Key type .llcrypt unregistered [ 5249.499852] Key type ._llcrypt unregistered [ 5275.490475] Key type ._llcrypt registered [ 5275.495249] Key type .llcrypt registered [ 5275.603740] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5276.458404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5276.532369] alg: No test for adler32 (adler32-zlib) [ 5277.553276] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 5277.793054] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5279.439205] Key type lgssc registered [ 5280.784717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5289.837793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5289.867729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.354564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.041456] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5301.742695] LustreError: 184115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.779819] LustreError: 184115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5306.851915] LustreError: 184116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.101812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.358294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5313.870423] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5323.238170] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5330.425446] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5338.023549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.475338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.992754] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5356.181871] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5361.024069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5381.603931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.608891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.652560] Lustre: Skipped 1 previous similar message [ 5381.690580] Lustre: Skipped 1 previous similar message [ 5385.812486] Lustre: server umount lustre-OST0000 complete [ 5391.843616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.849829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.863032] Lustre: Skipped 1 previous similar message [ 5391.870765] Lustre: Skipped 1 previous similar message [ 5396.518897] Lustre: server umount lustre-MDT0000 complete [ 5396.961241] LustreError: 184115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.988141] LustreError: 184115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5400.390499] LustreError: 185960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797549 with bad export cookie 4992978028463312435 [ 5400.392050] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5400.398161] LustreError: 185960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5400.758945] Lustre: server umount lustre-MDT0001 complete [ 5413.471674] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5416.506558] Key type lgssc unregistered [ 5416.840586] LNet: 187673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5416.855584] LNetError: 187673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5416.877237] LNet: Removed LNI 192.168.206.138@tcp [ 5417.777176] Key type .llcrypt unregistered [ 5417.778782] Key type ._llcrypt unregistered [ 5442.690796] Key type ._llcrypt registered [ 5442.696660] Key type .llcrypt registered [ 5442.921209] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5444.170412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5444.202176] alg: No test for adler32 (adler32-zlib) [ 5445.640362] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 5445.870486] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5447.543231] Key type lgssc registered [ 5448.759988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5456.505726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5456.519188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.926081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.528382] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5468.130497] LustreError: 189087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.148236] LustreError: 189087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5471.331592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.823053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5476.261853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5483.975728] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5490.991828] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5498.319168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5498.835402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.964055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5505.085486] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5512.218370] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5523.436429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5523.451517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.470746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5525.090586] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 5526.501384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5526.507374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5526.526992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.190101] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 5530.198465] Lustre: Skipped 1 previous similar message [ 5535.314326] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 5535.320494] Lustre: Skipped 2 previous similar messages [ 5536.227783] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5536.472525] Lustre: server umount lustre-OST0000 complete [ 5543.874646] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff956c4c49c800.ost_server_uuid 50 [ 5545.525804] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5561.314374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.334814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5561.344290] Lustre: Skipped 1 previous similar message [ 5566.463167] LustreError: 189815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.480628] LustreError: 189815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5566.564621] Lustre: server umount lustre-MDT0000 complete [ 5570.816348] LustreError: 189068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797719 with bad export cookie 9091528266691313781 [ 5570.830652] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.121678] Lustre: server umount lustre-MDT0001 complete [ 5581.075312] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:28:48 (1787797728) [ 5588.364574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.833436] LustreError: 193003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.901330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.406989] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5594.083258] LustreError: 193004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.200235] LustreError: 193003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.881482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.317137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5607.829578] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5614.693357] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5620.615944] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5627.736863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5628.131487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.708295] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5634.294982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5641.042261] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5648.072285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5648.264465] Lustre: lustre-OST0001: new disk, initializing [ 5648.267605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5648.276949] Lustre: 195842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5648.389140] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5654.621073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5654.631663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5654.717401] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5654.718912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5664.500695] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.377050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.379953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.384134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.390699] LustreError: Skipped 1 previous similar message [ 5670.425178] Lustre: Skipped 2 previous similar messages [ 5673.777225] Lustre: server umount lustre-OST0000 complete [ 5675.495784] LustreError: 194823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.518913] LustreError: 194823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5680.465353] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5680.678911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5686.279433] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5686.555644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5695.979210] LustreError: 196652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.998081] LustreError: 196652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5701.103587] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.106358] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.141805] Lustre: Skipped 1 previous similar message [ 5706.215395] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5706.224547] Lustre: Skipped 5 previous similar messages [ 5707.119125] Lustre: server umount lustre-OST0001 complete [ 5711.330507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.345600] Lustre: Skipped 1 previous similar message [ 5717.474711] Lustre: server umount lustre-MDT0000 complete [ 5722.101321] LustreError: 192984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797871 with bad export cookie 9091528266691314852 [ 5722.115407] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.119224] LustreError: 192984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5722.423901] Lustre: server umount lustre-MDT0001 complete [ 5747.826471] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:31:35 (1787797895) [ 5754.120795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.376293] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5754.409778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5754.684305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.191842] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5768.565343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.694357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5768.745449] Lustre: 199291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5773.808224] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5781.797592] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5787.996298] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5796.487568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5796.883778] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5804.456095] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5805.748289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5813.125725] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5819.892283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.952780] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5820.127472] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.137663] Lustre: Skipped 2 previous similar messages [ 5826.419786] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5827.731407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5834.034746] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.332228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.338769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.339579] Lustre: Skipped 1 previous similar message [ 5838.351460] Lustre: Skipped 5 previous similar messages [ 5843.435125] LustreError: 201141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.463865] LustreError: 201141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5843.706330] Lustre: server umount lustre-OST0000 complete [ 5863.396363] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5867.025449] Lustre: server umount lustre-OST0001 complete [ 5871.585069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5871.600139] LustreError: Skipped 1 previous similar message [ 5871.605186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.637499] Lustre: Skipped 2 previous similar messages [ 5877.561837] Lustre: server umount lustre-MDT0000 complete [ 5881.532803] LustreError: 199285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798030 with bad export cookie 9091528266691316259 [ 5881.536246] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.549964] LustreError: 199285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5881.876098] Lustre: server umount lustre-MDT0001 complete [ 5908.787106] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:34:15 (1787798055) [ 5925.252663] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5934.526441] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5970.360846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.693240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5970.743274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5970.877960] Lustre: lustre-MDT0000: new disk, initializing [ 5970.956641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.968992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5975.244501] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5985.979533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.101835] Lustre: 207383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5986.157995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5986.164338] Lustre: Skipped 1 previous similar message [ 5986.232889] Lustre: lustre-MDT0001: new disk, initializing [ 5986.336420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5986.350747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5990.587763] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6001.294376] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6007.610832] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6015.581880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.962594] Lustre: lustre-OST0000: new disk, initializing [ 6015.969836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6015.983327] Lustre: 209351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6022.374704] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6023.728438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6023.750408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6023.864621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6032.013588] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6036.905740] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6037.090594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6041.429638] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6041.594763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6044.644899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.654764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.662156] Lustre: Skipped 1 previous similar message [ 6044.663615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.671252] LustreError: Skipped 1 previous similar message [ 6044.688961] Lustre: Skipped 7 previous similar messages [ 6049.604045] Lustre: server umount lustre-OST0000 complete [ 6059.479357] Lustre: server umount lustre-MDT0000 complete [ 6060.002212] LustreError: 207396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.032513] LustreError: 207396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6062.955474] LustreError: 207376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798212 with bad export cookie 9091528266691318044 [ 6062.962583] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.966401] LustreError: 207376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6063.347824] Lustre: server umount lustre-MDT0001 complete [ 6072.381212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.212682] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6085.183309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.929749] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6097.712685] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6103.253856] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6110.877045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.271461] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6126.382836] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6131.927247] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6132.192742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6137.375087] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6137.679070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6141.920519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.891566] Lustre: server umount lustre-OST0000 complete [ 6152.724530] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6158.191432] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6187.544562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.053951] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.077419] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6195.267131] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6203.529618] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.741233] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6209.450165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.477675] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6214.629387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6214.630799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.637326] LustreError: Skipped 1 previous similar message [ 6214.655803] Lustre: Skipped 5 previous similar messages [ 6225.377327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6225.540675] Lustre: server umount lustre-OST0000 complete [ 6235.704749] Lustre: server umount lustre-MDT0000 complete [ 6239.609796] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798388 with bad export cookie 9091528266691318919 [ 6239.614572] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6239.625975] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6239.949819] Lustre: server umount lustre-MDT0001 complete [ 6249.948132] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:39:57 (1787798397) [ 6257.074648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.797403] LustreError: 217538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.835089] LustreError: 217538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6257.942470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6257.954649] Lustre: Skipped 6 previous similar messages [ 6263.271757] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6272.456587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.641413] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6287.786659] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6294.144297] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6301.294597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.526426] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6316.754355] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6322.511859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6322.762733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6327.705815] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6327.984108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6336.197041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.411744] Lustre: lustre-OST0001: new disk, initializing [ 6336.423097] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6336.432244] Lustre: 220911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6342.792804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6342.819930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6342.957933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6345.020187] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6356.698694] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6362.749148] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6363.000896] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6368.540341] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6368.730097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6373.857364] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6373.878510] LustreError: Skipped 1 previous similar message [ 6373.890151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6373.904165] Lustre: Skipped 17 previous similar messages [ 6385.119320] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6385.291550] Lustre: server umount lustre-OST0001 complete [ 6392.363282] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6392.655977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6397.999532] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6398.217870] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6427.199327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.756448] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6432.777690] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6432.778108] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6433.347499] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6439.618141] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6441.128378] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6449.312492] Lustre: server umount lustre-OST0001 complete [ 6480.351221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6480.603694] Lustre: server umount lustre-OST0000 complete [ 6488.032480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.052762] Lustre: Skipped 9 previous similar messages [ 6490.790176] Lustre: server umount lustre-MDT0000 complete [ 6495.097930] LustreError: 217518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798644 with bad export cookie 9091528266691319892 [ 6495.099506] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6495.114175] LustreError: 217518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6501.598675] Lustre: server umount lustre-MDT0001 complete [ 6530.870664] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:44:37 (1787798677) [ 6538.117486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.439490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6538.470130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6543.897228] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6553.892682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.043603] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6554.073349] Lustre: 226159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6559.138483] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6567.176401] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6573.678586] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6580.630316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6580.861397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6588.119674] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6595.222061] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6604.101709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6604.145868] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6610.161524] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6618.375935] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6620.146936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6624.973679] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6625.264522] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6630.633797] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6630.915647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6638.363203] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956c478b4800.ost_server_uuid 50 [ 6639.805714] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956c478b4800.ost_server_uuid in IDLE state after 0 sec [ 6640.746807] Lustre: Permanently deactivating lustre-OST0001 [ 6640.775367] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6644.309471] Lustre: Permanently reactivating lustre-OST0001 [ 6652.386420] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.406546] LustreError: Skipped 6 previous similar messages [ 6655.640506] Lustre: server umount lustre-OST0001 complete [ 6657.515643] LustreError: 228007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.541472] LustreError: 228007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6665.947266] Lustre: server umount lustre-OST0000 complete [ 6684.127226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6684.370694] Lustre: server umount lustre-MDT0000 complete [ 6688.352656] LustreError: 226151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798837 with bad export cookie 9091528266691321180 [ 6688.376495] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.642219] Lustre: server umount lustre-MDT0001 complete [ 6711.416086] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:47:38 (1787798858) [ 6719.233439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.598484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6719.641918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6719.662699] Lustre: Skipped 5 previous similar messages [ 6724.687860] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6733.191843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.267404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6733.341338] Lustre: 232853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6737.437472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6744.427952] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6749.937885] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6756.646460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.890195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6756.910378] Lustre: Permanently deactivating lustre-OST0000 [ 6756.936551] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6763.683778] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6766.623594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6771.802263] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6778.807438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.880562] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6779.070129] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6779.078250] Lustre: Skipped 11 previous similar messages [ 6784.703811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6785.263931] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6792.350385] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.251557] Lustre: Permanently reactivating lustre-OST0000 [ 6794.274059] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6794.286032] Lustre: Skipped 2 previous similar messages [ 6794.591265] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6794.599442] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6794.613314] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6794.622558] Lustre: Skipped 1 previous similar message [ 6794.661450] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6794.678927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6814.481239] Lustre: server umount lustre-OST0001 complete [ 6824.673143] Lustre: server umount lustre-OST0000 complete [ 6834.520706] Lustre: server umount lustre-MDT0000 complete [ 6838.759178] LustreError: 232846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798987 with bad export cookie 9091528266691322860 [ 6838.770810] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.775712] LustreError: 232846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6839.157565] Lustre: server umount lustre-MDT0001 complete [ 6850.503753] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6853.836645] Key type lgssc unregistered [ 6854.104138] LNet: 237730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6854.118182] LNetError: 237730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6854.136621] LNet: Removed LNI 192.168.206.138@tcp [ 6855.069724] Key type .llcrypt unregistered [ 6855.073786] Key type ._llcrypt unregistered [ 6879.910307] Key type ._llcrypt registered [ 6879.912821] Key type .llcrypt registered [ 6879.982294] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6880.964213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.978696] alg: No test for adler32 (adler32-zlib) [ 6882.026608] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 6882.339515] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6884.057123] Key type lgssc registered [ 6885.384935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.609983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6896.630454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.081722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.238891] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6908.389128] LustreError: 239199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.408962] LustreError: 239199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6909.979633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.098810] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6910.119262] Lustre: 239188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6910.130590] Lustre: Permanently deactivating lustre-MDT0001 [ 6910.140288] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6910.437345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.550511] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6916.105078] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6922.269853] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6927.272641] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6935.543120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.264558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.424061] LustreError: 241039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6943.449623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6945.887088] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6948.849504] LustreError: 241039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6953.969788] LustreError: 241041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.010864] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6955.094673] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.110692] LustreError: 241332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.134136] LustreError: 241332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6960.841994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6961.076425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.271469] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6966.274256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6966.535937] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6973.709088] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6977.527917] Lustre: Permanently reactivating lustre-MDT0001 [ 6977.554275] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6977.564663] Lustre: Skipped 2 previous similar messages [ 6982.280783] Lustre: lustre-MDT0001: Client 44557817-f962-4951-ab2b-36fee6959740 (at 192.168.206.38@tcp) reconnecting [ 6983.774832] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.785327] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6983.790609] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6983.797583] LustreError: 242764:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6983.800887] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6983.806637] LustreError: 242764:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6983.806757] LustreError: 242764:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.138@tcp: cfg command failed: rc = -108 [ 6983.806778] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6983.806778] [ 6983.807329] LustreError: 239190:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6988.431341] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.987351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6995.185638] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6995.378217] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6999.359921] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6999.501368] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7001.813430] Lustre: Permanently deactivating lustre-MDT0001 [ 7001.858619] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7001.865752] Lustre: Skipped 2 previous similar messages [ 7008.859059] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7017.953557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7017.959220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.971252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.566845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.570401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.592558] Lustre: Skipped 2 previous similar messages [ 7027.685953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7027.699039] Lustre: Skipped 6 previous similar messages [ 7032.287331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7032.473515] Lustre: server umount lustre-MDT0000 complete [ 7032.801759] LustreError: 239194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.822061] LustreError: 239194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7036.221975] LustreError: 243684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799185 with bad export cookie 11028164435956241864 [ 7036.229981] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.234540] LustreError: 243684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7036.557512] Lustre: server umount lustre-MDT0001 complete [ 7042.396658] Lustre: server umount lustre-OST0000 complete [ 7053.852969] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:53:20 (1787799200) [ 7063.575178] Lustre: server umount lustre-OST0001 complete [ 7070.330915] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7079.390724] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7114.386995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.637176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7114.649716] Lustre: Skipped 2 previous similar messages [ 7114.669865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7114.746105] Lustre: lustre-MDT0000: new disk, initializing [ 7114.796571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.806281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7119.579322] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7130.374305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.519244] Lustre: 247478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7130.595393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7130.598926] Lustre: Skipped 1 previous similar message [ 7130.698393] Lustre: lustre-MDT0001: new disk, initializing [ 7130.808210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.847157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7130.854644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7135.197710] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7145.301461] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7150.747450] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7157.085616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.293208] Lustre: lustre-OST0000: new disk, initializing [ 7157.296402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7157.301401] Lustre: 249447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7157.379066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.838901] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7166.007936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7166.015345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7166.136687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7174.168627] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7176.598951] LustreError: 247486:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7185.492330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.688706] Lustre: lustre-OST0001: new disk, initializing [ 7185.698838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7185.710203] Lustre: 250561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7186.667741] LustreError: 247486:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7186.688991] LustreError: 247485:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7191.933764] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7193.219719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7193.244911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7193.359689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7196.760617] LustreError: 247485:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7201.683729] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7203.180816] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7206.503543] Lustre: lustre-OST0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 7208.933140] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.938105] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.953029] LustreError: Skipped 1 previous similar message [ 7208.979325] Lustre: Skipped 1 previous similar message [ 7211.605475] Lustre: lustre-OST0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 7211.615796] Lustre: Skipped 2 previous similar messages [ 7219.173354] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7219.286877] Lustre: server umount lustre-OST0001 complete [ 7221.837462] LustreError: 249432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.113778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7238.121162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.136840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7238.144237] Lustre: Skipped 6 previous similar messages [ 7241.989642] Lustre: server umount lustre-OST0000 complete [ 7248.864872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7248.871657] LustreError: Skipped 1 previous similar message [ 7248.876423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.893089] Lustre: Skipped 1 previous similar message [ 7251.607842] Lustre: server umount lustre-MDT0000 complete [ 7255.011816] LustreError: 247485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.037593] LustreError: 247485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7255.145945] LustreError: 247470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799404 with bad export cookie 11028164435956244678 [ 7255.147408] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.157397] LustreError: 247470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7261.495809] Lustre: server umount lustre-MDT0001 complete [ 7270.049732] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7272.440306] Key type lgssc unregistered [ 7272.689484] LNet: 252599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7272.708679] LNetError: 252599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7272.727977] LNet: Removed LNI 192.168.206.138@tcp [ 7273.498814] Key type .llcrypt unregistered [ 7273.502203] Key type ._llcrypt unregistered [ 7309.421814] Key type ._llcrypt registered [ 7309.426156] Key type .llcrypt registered [ 7309.540706] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7310.445563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7310.618090] alg: No test for adler32 (adler32-zlib) [ 7311.623355] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 7311.905863] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7313.575280] Key type lgssc registered [ 7314.583823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.915197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7321.937639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.183220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7323.207181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7323.320051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.142723] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7334.199948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.316370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7334.350495] Lustre: 254803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7334.689658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7338.247792] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7345.090344] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7350.962696] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7356.720730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7356.858433] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7357.037658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.580597] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7363.166212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7369.020367] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7424.492816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.516987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7424.535581] Lustre: Skipped 1 previous similar message [ 7426.853570] Lustre: server umount lustre-OST0000 complete [ 7431.811544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7444.444717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7444.688767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.093741] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7458.961935] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7464.443277] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7464.463594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7464.474397] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7464.498830] Lustre: Skipped 1 previous similar message [ 7500.264319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7500.273761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.294821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7500.300809] Lustre: Skipped 2 previous similar messages [ 7504.556790] Lustre: server umount lustre-OST0000 complete [ 7508.973791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7508.988427] LustreError: Skipped 1 previous similar message [ 7508.994498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.015554] Lustre: Skipped 1 previous similar message [ 7509.025082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.335826] Lustre: server umount lustre-MDT0000 complete [ 7518.073080] LustreError: 254795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799667 with bad export cookie 745198077058333201 [ 7518.077647] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7518.083037] LustreError: 254795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7518.434842] Lustre: server umount lustre-MDT0001 complete [ 7529.086967] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7532.836547] Key type lgssc unregistered [ 7533.245636] LNet: 260235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7533.257599] LNetError: 260235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7533.274939] LNet: Removed LNI 192.168.206.138@tcp [ 7534.506789] Key type .llcrypt unregistered [ 7534.518075] Key type ._llcrypt unregistered [ 7560.834080] Key type ._llcrypt registered [ 7560.838170] Key type .llcrypt registered [ 7560.964154] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7561.845795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7561.890114] alg: No test for adler32 (adler32-zlib) [ 7562.955395] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 7563.191389] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7564.872476] Key type lgssc registered [ 7566.188276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7573.647312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7573.689435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.194810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.796732] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7585.769158] LustreError: 261668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.787975] LustreError: 261668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7588.050309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7588.451946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7593.267375] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7600.400767] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7606.118943] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.334902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.631489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.089654] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7620.860856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7625.187363] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7649.761616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.765958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.777424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7651.809294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7651.823460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7651.846211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.301255] Lustre: server umount lustre-OST0000 complete [ 7660.001175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7660.011190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.034704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.044239] Lustre: Skipped 1 previous similar message [ 7662.051114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7662.051159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.168920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7667.175991] Lustre: Skipped 2 previous similar messages [ 7672.287130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7672.534266] Lustre: server umount lustre-MDT0000 complete [ 7676.230868] LustreError: 261649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799825 with bad export cookie 1390116424623153491 [ 7676.237381] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.243739] LustreError: 261649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7676.717838] Lustre: server umount lustre-MDT0001 complete [ 7687.256645] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7689.989436] Key type lgssc unregistered [ 7690.332054] LNet: 265975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7690.344591] LNetError: 265975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7690.360891] LNet: Removed LNI 192.168.206.138@tcp [ 7691.494130] Key type .llcrypt unregistered [ 7691.496118] Key type ._llcrypt unregistered [ 7709.578800] Key type ._llcrypt registered [ 7709.584692] Key type .llcrypt registered [ 7709.740522] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7710.586931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7710.611101] alg: No test for adler32 (adler32-zlib) [ 7711.679696] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 7711.941741] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7713.607681] Key type lgssc registered [ 7714.470370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7722.450316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7722.477509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.880110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.589721] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7734.246808] LustreError: 267197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.271224] LustreError: 267197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7737.645798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.121766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7743.242265] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7750.600501] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7757.612884] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7764.892107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.450735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.540752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7773.002046] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7780.884167] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.174668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.188234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.211609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.244040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.247037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.276514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.970690] Lustre: server umount lustre-OST0000 complete [ 7808.494899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7808.504724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.506749] Lustre: Skipped 1 previous similar message [ 7808.526521] Lustre: Skipped 2 previous similar messages [ 7811.314951] Lustre: server umount lustre-MDT0000 complete [ 7813.600565] LustreError: 267192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.619651] LustreError: 267192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7815.450851] LustreError: 267179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799964 with bad export cookie 17689607655381837593 [ 7815.456172] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.464891] LustreError: 267179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7816.141548] Lustre: server umount lustre-MDT0001 complete [ 7828.123849] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7830.863792] Key type lgssc unregistered [ 7831.168357] LNet: 270877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7831.178121] LNetError: 270877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7831.211956] LNet: Removed LNI 192.168.206.138@tcp [ 7832.163642] Key type .llcrypt unregistered [ 7832.169984] Key type ._llcrypt unregistered [ 7856.187418] Key type ._llcrypt registered [ 7856.193445] Key type .llcrypt registered [ 7856.284182] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7857.261305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7857.333031] alg: No test for adler32 (adler32-zlib) [ 7858.433462] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 7858.709092] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7860.383205] Key type lgssc registered [ 7861.627846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7868.687110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7868.701480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.150385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.273963] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7880.677697] LustreError: 272292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.702705] LustreError: 272292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7881.831090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.192572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.536882] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7893.869925] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7898.793285] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7905.073455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7905.355294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.415385] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7915.648897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7918.556277] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7943.651095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.660620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.670401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.726220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.735579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.769984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7949.233462] Lustre: server umount lustre-OST0000 complete [ 7953.890698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7953.900775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.934037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.939421] Lustre: Skipped 1 previous similar message [ 7956.963987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7956.966872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.083189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.096041] Lustre: Skipped 2 previous similar messages [ 7968.229313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7968.393773] Lustre: server umount lustre-MDT0000 complete [ 7971.800598] LustreError: 272274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800120 with bad export cookie 5218944072368305259 [ 7971.804925] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7971.814185] LustreError: 272274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7972.193081] Lustre: server umount lustre-MDT0001 complete [ 7982.462387] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7985.567473] Key type lgssc unregistered [ 7985.921739] LNet: 276597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7985.930249] LNetError: 276597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7986.985072] LNet: Removed LNI 192.168.206.138@tcp [ 7987.939975] Key type .llcrypt unregistered [ 7987.941735] Key type ._llcrypt unregistered [ 8003.832238] Key type ._llcrypt registered [ 8003.834038] Key type .llcrypt registered [ 8003.933123] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8005.076876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8005.097802] alg: No test for adler32 (adler32-zlib) [ 8006.101991] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 8006.337601] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8008.040069] Key type lgssc registered [ 8009.025765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8016.383677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8016.405030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.927779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.179888] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8028.128981] LustreError: 277829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.145324] LustreError: 277829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8029.460602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8029.717704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8033.590213] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8040.184873] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8045.530811] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8052.736782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.135205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8058.944487] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8063.425861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8065.838381] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8078.808864] Lustre: server umount lustre-OST0000 complete [ 8083.952255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.955434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8083.960464] Lustre: Skipped 1 previous similar message [ 8083.969368] Lustre: Skipped 1 previous similar message [ 8089.058967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.064849] Lustre: Skipped 1 previous similar message [ 8092.220481] Lustre: server umount lustre-MDT0000 complete [ 8094.111114] Lustre: 277124:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787800227/real 1787800227] req@000000005fb8486a x1874644335356800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787800243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8094.148570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.164984] LustreError: 277830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8094.177749] LustreError: 277830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8095.532918] LustreError: 277795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800244 with bad export cookie 2759934768130078857 [ 8095.541275] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.544831] LustreError: 277795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8095.858799] Lustre: server umount lustre-MDT0001 complete [ 8106.456811] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8108.895279] Key type lgssc unregistered [ 8109.169991] LNet: 281526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8109.177620] LNetError: 281526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8109.188036] LNet: Removed LNI 192.168.206.138@tcp [ 8109.916986] Key type .llcrypt unregistered [ 8109.921858] Key type ._llcrypt unregistered [ 8121.015380] Key type ._llcrypt registered [ 8121.017912] Key type .llcrypt registered [ 8121.101328] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8138.611551] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 8151.352510] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8152.389637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8152.420109] alg: No test for adler32 (adler32-zlib) [ 8153.441808] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 8153.789516] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8155.463705] Key type lgssc registered [ 8156.500599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8190.724409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8190.771690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.997943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8192.026872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8192.148270] Lustre: lustre-MDT0000: new disk, initializing [ 8192.232103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.254788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8196.668852] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8206.663722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.778670] Lustre: 285058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8206.831752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8206.834907] Lustre: Skipped 1 previous similar message [ 8206.898669] Lustre: lustre-MDT0001: new disk, initializing [ 8206.979819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8207.022550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8207.032849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8210.572823] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8218.616677] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8223.222799] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8228.623307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8228.892672] Lustre: lustre-OST0000: new disk, initializing [ 8228.895641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8228.901992] Lustre: 287024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8228.979050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.389890] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8236.620352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8236.631886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8236.724789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8242.927737] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8246.760540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8246.950618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8250.904886] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8251.159118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8257.513808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8257.525452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.537943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.547220] Lustre: Skipped 1 previous similar message [ 8262.627456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8267.232231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.343204] Lustre: server umount lustre-OST0000 complete [ 8272.864400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.868130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.880452] Lustre: Skipped 2 previous similar messages [ 8272.888304] Lustre: Skipped 1 previous similar message [ 8277.018779] Lustre: server umount lustre-MDT0000 complete [ 8280.450360] LustreError: 285049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800429 with bad export cookie 1659746199211512197 [ 8280.457974] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8280.462374] LustreError: 285049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8280.800001] Lustre: server umount lustre-MDT0001 complete [ 8287.732370] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:13:55 (1787800435) [ 8293.105917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.469444] LustreError: 289436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.482543] LustreError: 289436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8293.568816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.103546] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8298.980169] LustreError: 289437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.090236] LustreError: 289436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.921162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.330860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.328354] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8315.245246] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.923979] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8325.519747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8325.794460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.270685] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8337.889710] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8348.640523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8348.647582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.651669] LustreError: Skipped 1 previous similar message [ 8348.657700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.661157] Lustre: Skipped 1 previous similar message [ 8348.669127] Lustre: Skipped 1 previous similar message [ 8353.765473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8353.772916] Lustre: Skipped 2 previous similar messages [ 8354.662437] Lustre: server umount lustre-OST0000 complete [ 8358.880495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.002151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.011711] Lustre: Skipped 3 previous similar messages [ 8365.021822] Lustre: server umount lustre-MDT0000 complete [ 8368.743883] LustreError: 289416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800517 with bad export cookie 1659746199211513079 [ 8368.745773] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8368.762129] LustreError: 289416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8369.098708] Lustre: server umount lustre-MDT0001 complete [ 8380.508561] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8382.797425] Key type lgssc unregistered [ 8383.044608] LNet: 292972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8383.048910] LNetError: 292972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8383.060574] LNet: Removed LNI 192.168.206.138@tcp [ 8383.882578] Key type .llcrypt unregistered [ 8383.885589] Key type ._llcrypt unregistered [ 8408.614681] Key type ._llcrypt registered [ 8408.617616] Key type .llcrypt registered [ 8408.705593] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 8421.281390] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8422.280297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8422.352164] alg: No test for adler32 (adler32-zlib) [ 8423.418523] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 8423.685961] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8425.391165] Key type lgssc registered [ 8426.419371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8467.129272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.156612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.368216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8468.396246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8468.496112] Lustre: lustre-MDT0000: new disk, initializing [ 8468.577949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8468.599402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8472.538696] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8482.838247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8482.977074] Lustre: 296772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8483.044713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8483.049272] Lustre: Skipped 1 previous similar message [ 8483.132919] Lustre: lustre-MDT0001: new disk, initializing [ 8483.231932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8483.286776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8483.298760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8487.329869] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8496.354662] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8501.183188] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8507.705437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8507.995489] Lustre: lustre-OST2710: new disk, initializing [ 8507.999147] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8508.010960] Lustre: 298743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8508.153810] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8514.348574] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8517.691949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8517.724060] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8517.837947] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8522.894621] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8529.221956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8529.387048] Lustre: lustre-OST03e8: new disk, initializing [ 8529.391307] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8529.398131] Lustre: 299806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8529.513268] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8536.007220] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8538.715453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8538.724749] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8538.804517] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8543.269176] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8551.274304] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8551.542949] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8555.810430] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8556.012268] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8560.434932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8560.712243] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8565.568804] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8565.770332] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8572.765178] Lustre: server umount lustre-MDT0000 complete [ 8574.953488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8576.256259] LustreError: 296765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800725 with bad export cookie 17265246904414711912 [ 8576.260019] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.265320] LustreError: 296765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8576.620487] Lustre: server umount lustre-MDT0001 complete [ 8590.489111] Lustre: server umount lustre-OST2710 complete [ 8603.775198] Lustre: server umount lustre-OST03e8 complete [ 8618.384354] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 8626.877790] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8660.441520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.700902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8660.729527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8660.810221] Lustre: lustre-MDT0000: new disk, initializing [ 8660.926761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8660.947913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8664.650378] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8674.204422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.315702] Lustre: 305052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8674.386263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8674.398681] Lustre: Skipped 1 previous similar message [ 8674.454670] Lustre: lustre-MDT0001: new disk, initializing [ 8674.564733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8674.633879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8674.649383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8678.409816] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8686.434340] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8691.472951] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8697.601690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8697.852169] Lustre: lustre-OST0000: new disk, initializing [ 8697.862745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8697.874109] Lustre: 307019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8697.966506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.508085] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8707.695314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8707.701915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8707.826223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8713.848083] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8718.670964] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8718.967528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8723.694522] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8723.913040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8728.548391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8728.553390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.559637] LustreError: Skipped 1 previous similar message [ 8728.580846] Lustre: Skipped 3 previous similar messages [ 8728.588613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8728.599455] Lustre: Skipped 1 previous similar message [ 8733.666298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8733.675267] Lustre: Skipped 1 previous similar message [ 8738.787583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8738.801446] Lustre: Skipped 1 previous similar message [ 8740.319489] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8740.625308] Lustre: server umount lustre-OST0000 complete [ 8746.470256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.476748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.484730] Lustre: Skipped 1 previous similar message [ 8746.489210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.028667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.862990] Lustre: server umount lustre-MDT0000 complete [ 8754.145175] LustreError: 305064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.160483] LustreError: 305064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8754.877956] LustreError: 305044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800904 with bad export cookie 17265246904414713592 [ 8754.880183] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8754.888228] LustreError: 305044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8755.337907] Lustre: server umount lustre-MDT0001 complete [ 8764.598782] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:21:51 (1787800911) [ 8766.092258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8768.196653] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:21:55 (1787800915) [ 8772.136124] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8799.459551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.792571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8799.810550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8800.020496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.273199] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8812.695289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.858746] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8812.902903] Lustre: 310877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8817.323671] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8838.644966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8838.887267] Lustre: Denying initial registration attempt from nid 192.168.206.138@tcp, specified as failover [ 8838.894456] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8838.900252] LustreError: 312332:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8838.907639] LustreError: 312332:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8838.911558] LustreError: 312332:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8838.986897] Lustre: server umount lustre-OST0000 complete [ 8838.989881] LustreError: 312332:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8845.284055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.287023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8845.299668] Lustre: Skipped 1 previous similar message [ 8845.318641] Lustre: Skipped 3 previous similar messages [ 8851.351047] Lustre: server umount lustre-MDT0000 complete [ 8854.723540] LustreError: 310869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801003 with bad export cookie 17265246904414714474 [ 8854.724775] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.734409] LustreError: 310869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8855.096324] Lustre: server umount lustre-MDT0001 complete [ 8864.646066] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8867.060889] Key type lgssc unregistered [ 8867.432110] LNet: 313451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8867.441823] LNetError: 313451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8867.459039] LNet: Removed LNI 192.168.206.138@tcp [ 8868.238760] Key type .llcrypt unregistered [ 8868.240299] Key type ._llcrypt unregistered [ 8879.182090] Key type ._llcrypt registered [ 8879.187902] Key type .llcrypt registered [ 8879.284840] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8880.521767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8880.557814] alg: No test for adler32 (adler32-zlib) [ 8881.657695] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 8881.931038] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8883.687817] Key type lgssc registered [ 8884.920101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8937.589211] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8949.563972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8949.614694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.868645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8950.887576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8951.072772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.601809] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8964.224667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8964.307918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8964.335916] Lustre: 316216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8964.569264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8968.480238] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8975.337967] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8981.694975] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9002.105899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9002.329929] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9002.483266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.220152] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9016.389176] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9019.872680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.885617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.915211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9020.392934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.413805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9024.714704] Lustre: server umount lustre-OST0000 complete [ 9030.633574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.663542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.681816] Lustre: Skipped 3 previous similar messages [ 9034.527081] Lustre: server umount lustre-MDT0000 complete [ 9038.100605] LustreError: 316206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801187 with bad export cookie 15524281618124922014 [ 9038.117309] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.121316] LustreError: 316206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9038.445166] Lustre: server umount lustre-MDT0001 complete [ 9048.545560] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9051.214572] Key type lgssc unregistered [ 9051.475689] LNet: 319786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9051.488721] LNetError: 319786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9052.519430] LNet: Removed LNI 192.168.206.138@tcp [ 9053.541823] Key type .llcrypt unregistered [ 9053.543798] Key type ._llcrypt unregistered [ 9082.275603] Key type ._llcrypt registered [ 9082.278789] Key type .llcrypt registered [ 9082.434461] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9084.234381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9084.316875] alg: No test for adler32 (adler32-zlib) [ 9085.568407] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 9085.844214] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9087.559315] Key type lgssc registered [ 9089.019402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9097.191397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9097.203020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.600201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9102.755765] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9107.946741] LustreError: 321199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.973061] LustreError: 321199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9111.349817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.725850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9115.994843] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9124.327994] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9130.201804] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9138.924651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.235477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9144.979384] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9152.811942] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9166.819244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.824140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9166.845618] Lustre: Skipped 1 previous similar message [ 9166.867811] Lustre: Skipped 2 previous similar messages [ 9170.024813] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 9171.938845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9171.946554] Lustre: Skipped 2 previous similar messages [ 9175.117358] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 9180.235339] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 9180.247643] Lustre: Skipped 3 previous similar messages [ 9181.151347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9181.341673] Lustre: server umount lustre-MDT0000 complete [ 9182.179807] LustreError: 321728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.201168] LustreError: 321728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9185.355189] LustreError: 323041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.421629] LustreError: 321179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801334 with bad export cookie 3479636368404964448 [ 9185.430084] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9185.432178] LustreError: 321179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9187.305671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.313036] Lustre: Skipped 1 previous similar message [ 9190.304214] LustreError: 321196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9190.304236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9190.317408] LustreError: 321196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9190.329571] Lustre: Skipped 1 previous similar message [ 9191.779170] Lustre: server umount lustre-MDT0001 complete [ 9197.815763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9206.462163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.861838] LustreError: 320507:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee5ac329 x1874645467832064/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9210.956768] LustreError: 324815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.988814] LustreError: 324815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9211.314782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.392130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9215.768949] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9220.521551] LustreError: 323032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.534259] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9220.540754] LustreError: 323032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9223.964220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.269607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9228.936912] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9229.292273] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9235.478744] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9237.094642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9242.521958] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9244.130409] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9249.772397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.779421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9249.792823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9249.792831] Lustre: Skipped 1 previous similar message [ 9255.168656] Lustre: server umount lustre-OST0000 complete [ 9260.015277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9260.017310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.038549] Lustre: Skipped 2 previous similar messages [ 9273.311532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9273.498226] Lustre: server umount lustre-MDT0000 complete [ 9275.362941] LustreError: 324821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.380807] LustreError: 324821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9276.835409] LustreError: 324008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801426 with bad export cookie 3479636368404971952 [ 9276.839443] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.852321] LustreError: 324008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9277.189853] Lustre: server umount lustre-MDT0001 complete [ 9288.525831] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9291.415254] Key type lgssc unregistered [ 9291.839931] LNet: 327489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9291.845662] LNetError: 327489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9291.860441] LNet: Removed LNI 192.168.206.138@tcp [ 9292.829597] Key type .llcrypt unregistered [ 9292.831916] Key type ._llcrypt unregistered [ 9317.230468] Key type ._llcrypt registered [ 9317.233940] Key type .llcrypt registered [ 9317.418872] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9318.759580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9318.806083] alg: No test for adler32 (adler32-zlib) [ 9319.887395] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 9320.173763] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9321.871228] Key type lgssc registered [ 9323.215351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9333.346828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9333.390657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.751281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.006314] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9343.971958] LustreError: 328903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.011749] LustreError: 328903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9349.095913] LustreError: 328904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.494835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.077478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9359.697597] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9369.000335] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9375.760424] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9379.750228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.769474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9379.779199] Lustre: Skipped 1 previous similar message [ 9384.928161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.933065] Lustre: Skipped 1 previous similar message [ 9390.053265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.061576] Lustre: Skipped 1 previous similar message [ 9393.119197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9393.309682] Lustre: server umount lustre-MDT0000 complete [ 9395.170337] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.200363] LustreError: 329633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9397.181893] LustreError: 328884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801546 with bad export cookie 11822134371962995984 [ 9397.192146] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9397.726663] Lustre: server umount lustre-MDT0001 complete [ 9412.996374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9413.400493] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9413.456022] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9413.809200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9418.770993] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9428.146994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.297684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9428.333387] Lustre: 331491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9428.693492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9432.615824] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9440.531298] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9447.010184] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9457.836991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.031066] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9458.041308] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9458.048842] LustreError: 333376:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9458.054763] LustreError: 333376:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9458.059521] LustreError: 333376:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9458.133310] Lustre: server umount lustre-OST0000 complete [ 9458.135599] LustreError: 333376:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9469.551865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9469.872785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9470.192619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.235730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9476.795885] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9485.820523] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9494.497620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9494.593883] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9494.656482] Lustre: lustre-OST0001: new disk, initializing [ 9494.659803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9494.667838] Lustre: 334917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9494.788496] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.723616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9500.728438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9500.828050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9500.862612] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9507.666571] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9510.887346] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9510.900501] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.916717] Lustre: Skipped 1 previous similar message [ 9510.926731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9511.407835] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9516.517325] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9516.525167] Lustre: Skipped 3 previous similar messages [ 9516.892740] Lustre: server umount lustre-OST0001 complete [ 9521.635191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.640111] LustreError: 333844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.642897] Lustre: Skipped 2 previous similar messages [ 9521.667585] LustreError: 333844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9526.629417] Lustre: server umount lustre-OST0000 complete [ 9531.360625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9531.370308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.395966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.406632] Lustre: Skipped 2 previous similar messages [ 9536.661370] Lustre: server umount lustre-MDT0000 complete [ 9536.994316] LustreError: 332560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9537.015892] LustreError: 332560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9540.332647] LustreError: 331483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801689 with bad export cookie 11822134371962996537 [ 9540.339442] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9540.340705] LustreError: 331483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9540.646824] Lustre: server umount lustre-MDT0001 complete [ 9551.197934] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9553.463768] Key type lgssc unregistered [ 9553.808909] LNet: 336874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9553.814277] LNetError: 336874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9553.838273] LNet: Removed LNI 192.168.206.138@tcp [ 9554.599387] Key type .llcrypt unregistered [ 9554.601555] Key type ._llcrypt unregistered [ 9626.316508] Key type ._llcrypt registered [ 9626.319410] Key type .llcrypt registered [ 9626.418562] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 9641.678435] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9644.012670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9644.062893] alg: No test for adler32 (adler32-zlib) [ 9645.389948] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 9645.746446] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9647.399643] Key type lgssc registered [ 9649.008492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9690.119626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9690.142448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.362985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9691.396306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9691.518582] Lustre: lustre-MDT0000: new disk, initializing [ 9691.681906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.722909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9697.420658] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9710.234896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.487288] Lustre: 341813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9710.587948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9710.591705] Lustre: Skipped 1 previous similar message [ 9710.679639] Lustre: lustre-MDT0001: new disk, initializing [ 9710.764592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9710.803520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9710.814138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9715.931584] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9727.102902] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.790477] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9741.876747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.155467] Lustre: lustre-OST0000: new disk, initializing [ 9742.164696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9742.178675] Lustre: 343780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9742.283658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.544197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9748.555609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9748.736273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9749.893895] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9761.940983] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9766.601269] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9766.895575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9773.063630] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9773.359897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9779.688121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9779.693445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.701620] LustreError: Skipped 1 previous similar message [ 9779.737684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.750503] Lustre: Skipped 1 previous similar message [ 9781.778574] Lustre: server umount lustre-OST0000 complete [ 9787.876715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9787.891704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.912341] Lustre: Skipped 1 previous similar message [ 9787.918824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9789.922534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9789.926335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9791.987935] Lustre: server umount lustre-MDT0000 complete [ 9796.190912] LustreError: 341803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801945 with bad export cookie 1905707214804994275 [ 9796.195153] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9796.203100] LustreError: 341803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9796.843743] Lustre: server umount lustre-MDT0001 complete [ 9805.756217] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:39:12 (1787801952) [ 9813.215955] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:39:20 (1787801960) [ 9822.243275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.647524] LustreError: 346536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.665384] LustreError: 346536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9822.715924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9827.494880] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9827.811141] LustreError: 346537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.906450] LustreError: 346536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9836.743629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.098088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9841.017608] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9847.401206] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9852.763780] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9859.256479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.558620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.055431] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9874.580734] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9882.220803] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9883.944734] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9885.630544] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9887.827484] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9889.507433] Lustre: Failing over lustre-MDT0000 [ 9890.005762] Lustre: server umount lustre-MDT0000 complete [ 9891.951302] LustreError: 346531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.348498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.351020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9893.366060] Lustre: Skipped 1 previous similar message [ 9897.036205] LustreError: 347798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.065337] LustreError: 347798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9907.293177] LustreError: 346532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.311992] LustreError: 346532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9908.477700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.603302] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9908.708813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 9908.717015] Lustre: Skipped 3 previous similar messages [ 9908.816892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9908.893504] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9912.402965] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9912.823464] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9913.836259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9913.871164] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9913.915295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9922.293958] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9924.002542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9927.489113] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9929.657881] Lustre: Failing over lustre-MDT0000 [ 9929.964383] Lustre: server umount lustre-MDT0000 complete [ 9933.436665] LustreError: 346532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.306919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.309716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9934.316935] Lustre: Skipped 2 previous similar messages [ 9948.360571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9948.480638] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.725465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9948.783777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9952.635419] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9953.766062] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9953.770202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9953.781987] Lustre: Skipped 2 previous similar messages [ 9953.897772] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9953.927268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9959.947237] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9961.410502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9974.243641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9974.249462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9974.262723] Lustre: Skipped 3 previous similar messages [ 9974.276903] Lustre: Skipped 2 previous similar messages [ 9977.147907] Lustre: server umount lustre-MDT0000 complete [ 9979.369858] LustreError: 346536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.400422] LustreError: 346536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9981.241075] LustreError: 346517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802130 with bad export cookie 1905707214804996984 [ 9981.241612] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.253481] LustreError: 346517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9981.566801] Lustre: server umount lustre-MDT0001 complete [ 9995.175857] Lustre: server umount lustre-OST0000 complete [10010.623612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.690987] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10024.352358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10024.694228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10024.699361] Lustre: Skipped 1 previous similar message [10029.517938] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10037.607141] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10043.124380] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10049.727930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.682730] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10056.180840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10063.100121] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10079.723535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.730526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10079.762704] Lustre: Skipped 1 previous similar message [10079.776402] Lustre: Skipped 1 previous similar message [10085.166840] Lustre: server umount lustre-OST0000 complete [10095.670252] Lustre: server umount lustre-MDT0000 complete [10100.088426] LustreError: 354718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802249 with bad export cookie 1905707214804997733 [10100.098422] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10100.107474] LustreError: 354718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10100.416449] Lustre: server umount lustre-MDT0001 complete [10111.176987] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10114.256821] Key type lgssc unregistered [10114.526540] LNet: 356430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10114.539483] LNetError: 356430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10114.570448] LNet: Removed LNI 192.168.206.138@tcp [10115.480457] Key type .llcrypt unregistered [10115.482878] Key type ._llcrypt unregistered [10142.302405] Key type ._llcrypt registered [10142.306229] Key type .llcrypt registered [10142.436728] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10143.222404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10143.282394] alg: No test for adler32 (adler32-zlib) [10144.337407] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [10144.521419] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10146.167177] Key type lgssc registered [10147.494548] Lustre: Echo OBD driver; http://www.lustre.org/ [10156.655728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10156.696325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.127601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10163.657335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10168.309401] LustreError: 357952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.341946] LustreError: 357952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10172.907824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.481375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10178.596216] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10188.291816] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10196.253681] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10205.795446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10206.645695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10215.659905] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10225.385566] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10228.156534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10242.300386] Lustre: server umount lustre-MDT0000 complete [10243.566064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.589557] LustreError: 357948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.591666] Lustre: Skipped 2 previous similar messages [10243.637841] LustreError: 357948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10247.020021] LustreError: 357934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802396 with bad export cookie 3470150260871896934 [10247.040972] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10247.045563] LustreError: 357934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10247.818157] Lustre: server umount lustre-MDT0001 complete [10262.241672] Lustre: server umount lustre-OST0000 complete [10277.747367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10278.242904] LustreError: 361689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.350689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10283.492342] LustreError: 361690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.526841] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10288.622900] LustreError: 361689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.072071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.590711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10297.373764] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10305.377067] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10311.694127] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10319.460224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10319.911548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.008954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10327.444198] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10337.931572] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10352.611502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.615247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.627141] Lustre: Skipped 2 previous similar messages [10352.637617] Lustre: Skipped 2 previous similar messages [10357.591871] Lustre: server umount lustre-MDT0000 complete [10357.740741] LustreError: 361686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.773634] LustreError: 361686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10361.997990] LustreError: 361670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802511 with bad export cookie 3470150260871898054 [10362.011336] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10362.024258] LustreError: 361670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10362.325766] Lustre: server umount lustre-MDT0001 complete [10377.480511] Lustre: server umount lustre-OST0000 complete [10394.873084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10395.208195] LustreError: 365601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.279562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10400.721103] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10410.525688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.432783] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10423.122571] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10428.696859] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.282600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.763802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10435.768788] Lustre: Skipped 1 previous similar message [10439.856305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10442.527589] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10450.134638] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10455.520395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10455.523763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.540119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10455.545844] Lustre: Skipped 1 previous similar message [10456.980625] Lustre: server umount lustre-OST0000 complete [10460.652780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10460.654367] LustreError: 357260:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000093f69477 x1874646577279104/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10460.714195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10460.720741] Lustre: Skipped 1 previous similar message [10462.176327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10465.760943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10465.772195] Lustre: Skipped 2 previous similar messages [10467.039161] Lustre: server umount lustre-MDT0000 complete [10470.860622] LustreError: 365581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802620 with bad export cookie 3470150260871899076 [10470.873433] LustreError: 365581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10470.880179] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10471.196398] Lustre: server umount lustre-MDT0001 complete [10482.353770] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10485.064457] Key type lgssc unregistered [10485.355373] LNet: 369130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10485.372840] LNetError: 369130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10485.396621] LNet: Removed LNI 192.168.206.138@tcp [10486.215301] Key type .llcrypt unregistered [10486.219390] Key type ._llcrypt unregistered [10513.406845] Key type ._llcrypt registered [10513.408929] Key type .llcrypt registered [10513.573930] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10514.656665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10514.837556] alg: No test for adler32 (adler32-zlib) [10515.970796] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [10516.288378] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10518.047279] Key type lgssc registered [10519.277295] Lustre: Echo OBD driver; http://www.lustre.org/ [10528.189344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10528.217128] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.229741] LustreError: 370569:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10528.274921] LustreError: 370569:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10528.286952] LustreError: 370569:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10528.297542] LustreError: 370569:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10528.308130] LustreError: 370569:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10538.064875] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.072896] LustreError: 370966:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10538.108940] LustreError: 370966:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10538.120183] LustreError: 370966:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10538.126257] LustreError: 370966:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10538.142930] LustreError: 370966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10559.695924] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10563.145946] Key type lgssc unregistered [10563.536529] LNet: 371970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10563.550893] LNetError: 371970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10563.573491] LNet: Removed LNI 192.168.206.138@tcp [10564.769402] Key type .llcrypt unregistered [10564.773096] Key type ._llcrypt unregistered [10583.558946] Key type ._llcrypt registered [10583.562728] Key type .llcrypt registered [10583.697397] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [10599.540753] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10600.948110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10601.063796] alg: No test for adler32 (adler32-zlib) [10602.168420] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [10602.422417] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10604.175188] Key type lgssc registered [10605.156747] Lustre: Echo OBD driver; http://www.lustre.org/ [10640.242898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10640.273718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.576216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10641.617138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10641.761409] Lustre: lustre-MDT0000: new disk, initializing [10641.846996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10641.879265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10646.158171] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10658.155071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10658.307382] Lustre: 375126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10658.387147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10658.393493] Lustre: Skipped 1 previous similar message [10658.506125] Lustre: lustre-MDT0001: new disk, initializing [10658.632359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10658.717559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10658.726365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10663.478538] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10677.763673] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10688.601605] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10698.776535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.942497] Lustre: lustre-OST0000: new disk, initializing [10698.947599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10698.953367] Lustre: 377095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10699.038791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10706.812424] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10708.630824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10708.653851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10708.742406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10717.969841] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10725.716619] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10726.071285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10731.460938] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10731.808908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10734.562115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10734.565865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10734.581848] LustreError: Skipped 1 previous similar message [10734.594445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10734.621860] Lustre: Skipped 1 previous similar message [10734.652946] Lustre: Skipped 1 previous similar message [10739.694149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10739.712435] Lustre: Skipped 1 previous similar message [10740.519221] Lustre: server umount lustre-OST0000 complete [10744.801396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.810971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.923282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.930822] Lustre: Skipped 2 previous similar messages [10750.819586] Lustre: server umount lustre-MDT0000 complete [10754.258775] LustreError: 375117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802903 with bad export cookie 8150830117260199808 [10754.276087] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10754.280355] LustreError: 375117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10754.638960] Lustre: server umount lustre-MDT0001 complete [10763.748296] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10765.454974] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:55:12 (1787802912) [10771.971603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.405966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.801207] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10777.572060] LustreError: 379557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.585062] LustreError: 379557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10782.693715] LustreError: 379558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.506113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.010868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.399665] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10796.940299] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10802.182628] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10808.942886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.307424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10816.974531] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10826.115938] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10832.947437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.111482] Lustre: lustre-OST0001: new disk, initializing [10833.120829] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10833.131444] Lustre: 382393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10833.232137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10839.663868] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10842.690140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10842.694938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10842.788548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10850.454626] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10858.085909] Lustre: lustre-OST0001: Not available for connect from 192.168.206.38@tcp (stopping) [10858.469598] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.474554] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10858.474808] Lustre: Skipped 1 previous similar message [10861.410346] Lustre: server umount lustre-OST0001 complete [10863.206644] LustreError: 381378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.707578] LustreError: 381378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.738798] LustreError: 381378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10873.841661] LustreError: 381380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.855453] LustreError: 381380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10878.959245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.968893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10878.978354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10878.999014] Lustre: Skipped 3 previous similar messages [10883.897251] Lustre: server umount lustre-OST0000 complete [10887.658660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10887.668331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.687457] Lustre: Skipped 1 previous similar message [10893.643943] Lustre: server umount lustre-MDT0000 complete [10894.314917] LustreError: 381761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.328272] LustreError: 381761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10896.882430] LustreError: 382392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803046 with bad export cookie 8150830117260200690 [10896.889282] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10896.895911] LustreError: 382392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10903.326584] Lustre: server umount lustre-MDT0001 complete [10914.030665] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10916.579901] Key type lgssc unregistered [10916.928305] LNet: 384381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10916.943934] LNetError: 384381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10916.969534] LNet: Removed LNI 192.168.206.138@tcp [10917.966174] Key type .llcrypt unregistered [10917.970602] Key type ._llcrypt unregistered [10971.464231] Key type ._llcrypt registered [10971.472165] Key type .llcrypt registered [10971.578527] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10973.092863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10973.143000] alg: No test for adler32 (adler32-zlib) [10974.297991] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [10974.628934] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10976.343862] Key type lgssc registered [10977.958368] Lustre: Echo OBD driver; http://www.lustre.org/ [10987.456500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10987.469239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10988.693567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10988.722454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10988.951085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10994.156695] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11003.255574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.328803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11003.353294] Lustre: 387176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11003.643465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11008.284630] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11014.892456] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11020.501303] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11024.352495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11024.369694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.383488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.915267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.922926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11032.039244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11032.050361] Lustre: Skipped 2 previous similar messages [11037.158221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11037.163412] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [11037.169472] Lustre: Skipped 1 previous similar message [11037.596602] Lustre: server umount lustre-MDT0000 complete [11042.522448] LustreError: 387168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803191 with bad export cookie 18170363630303993649 [11042.527151] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.532485] LustreError: 387168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11042.969264] Lustre: server umount lustre-MDT0001 complete [11054.446332] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:00:02 (1787803202) [11060.947981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.319646] LustreError: 389694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.342082] LustreError: 389694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11061.410704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11065.763876] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11066.852131] LustreError: 389695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.971679] LustreError: 389694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11073.659670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11073.914435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.620592] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11085.188762] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11091.084471] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11097.895994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.096731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11098.265320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11104.207128] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11111.572325] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.155457] Lustre: Permanently deactivating lustre-OST0000 [11121.171873] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11130.942227] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11133.279447] LustreError: 392739:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11133.288458] LustreError: 392739:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11133.298298] LustreError: 392739:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11136.228264] LustreError: 392787:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11136.232478] LustreError: 392787:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11144.645747] Lustre: server umount lustre-OST0000 complete [11147.471562] LustreError: 393037:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11147.493133] LustreError: 393037:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11150.816776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.835419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11150.836423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11150.877588] Lustre: Skipped 1 previous similar message [11159.130379] Lustre: server umount lustre-MDT0000 complete [11161.063470] LustreError: 390420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.069436] LustreError: 390420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11164.677026] LustreError: 389676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803313 with bad export cookie 18170363630303994223 [11164.680407] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.683156] LustreError: 389676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11174.127848] Lustre: server umount lustre-MDT0001 complete [11187.751423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.916765] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11196.031234] LustreError: 394198:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11202.460777] Lustre: 394538:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.138@tcp [11204.888681] Lustre: server umount MGS complete [11218.918780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.552258] LustreError: 395263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.580346] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11219.587553] Lustre: Skipped 1 previous similar message [11219.665643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11225.549152] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11230.177557] LustreError: 395264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.209967] LustreError: 395264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11235.203397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.542752] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11239.711262] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11247.299375] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11252.928478] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11254.740241] Lustre: Permanently reactivating lustre-OST0000 [11254.772977] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11254.783702] Lustre: Skipped 2 previous similar messages [11282.915390] LustreError: 397051:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11291.506507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.877174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11291.882153] Lustre: Skipped 1 previous similar message [11298.255149] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11307.274937] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11312.611911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11312.618896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11312.637235] Lustre: Skipped 1 previous similar message [11312.644740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11312.651420] Lustre: Skipped 2 previous similar messages [11317.762536] Lustre: server umount lustre-OST0000 complete [11322.852769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11322.859657] LustreError: Skipped 1 previous similar message [11322.866564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.879814] Lustre: Skipped 1 previous similar message [11327.920659] Lustre: server umount lustre-MDT0000 complete [11328.993614] LustreError: 395260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.017690] LustreError: 395260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11331.641577] LustreError: 395244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803480 with bad export cookie 18170363630303995280 [11331.648025] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11331.656062] LustreError: 395244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11332.002440] Lustre: server umount lustre-MDT0001 complete [11344.504397] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11348.654688] Key type lgssc unregistered [11349.250038] LNet: 399675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11349.284542] LNetError: 399675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11350.313256] LNet: Removed LNI 192.168.206.138@tcp [11351.499453] Key type .llcrypt unregistered [11351.501386] Key type ._llcrypt unregistered [11374.173617] Key type ._llcrypt registered [11374.176647] Key type .llcrypt registered [11374.269641] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [11386.899581] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11387.903206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11387.937698] alg: No test for adler32 (adler32-zlib) [11389.040482] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [11389.273390] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11390.951180] Key type lgssc registered [11391.906663] Lustre: Echo OBD driver; http://www.lustre.org/ [11437.140965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11437.196610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.426838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11438.483345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11438.568830] Lustre: lustre-MDT0000: new disk, initializing [11438.621207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11438.635291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11442.544624] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11453.043711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.117281] Lustre: 402813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11453.174581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11453.178234] Lustre: Skipped 1 previous similar message [11453.227464] Lustre: lustre-MDT0001: new disk, initializing [11453.296297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.322514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11453.329704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11458.564018] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11469.286324] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11475.160373] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11481.795452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11481.996193] Lustre: lustre-OST0000: new disk, initializing [11481.999297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11482.005378] Lustre: 404781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11482.082631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.539218] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11490.435177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11490.461508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11490.623368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11499.045734] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11504.478132] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11504.716423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11509.464736] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11509.711445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11516.392186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11516.393813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.412136] LustreError: Skipped 1 previous similar message [11516.419134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11516.487905] Lustre: Skipped 1 previous similar message [11519.839704] Lustre: server umount lustre-OST0000 complete [11526.625519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.628804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11526.656268] Lustre: Skipped 1 previous similar message [11526.668545] Lustre: Skipped 1 previous similar message [11531.745033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11531.775437] Lustre: Skipped 2 previous similar messages [11532.530995] Lustre: server umount lustre-MDT0000 complete [11536.807693] LustreError: 402803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803685 with bad export cookie 13954752032601372308 [11536.814738] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11536.822847] LustreError: 402803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11537.351497] Lustre: server umount lustre-MDT0001 complete [11549.947635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11551.997924] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:08:19 (1787803699) [11558.668085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.122662] LustreError: 407243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11559.146346] LustreError: 407243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11559.197625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11563.867759] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11564.517891] LustreError: 407244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11569.634229] LustreError: 407243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11573.075696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.431573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11579.833900] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11590.966212] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.864148] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11607.364253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.971894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11614.409841] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11624.240574] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11626.706059] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11633.625714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11639.782336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.784300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11639.796905] Lustre: Skipped 1 previous similar message [11644.910674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11644.924464] Lustre: Skipped 1 previous similar message [11645.872965] Lustre: server umount lustre-OST0000 complete [11650.025721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.043920] Lustre: Skipped 1 previous similar message [11655.157178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11655.168198] Lustre: Skipped 4 previous similar messages [11655.511555] Lustre: server umount lustre-MDT0000 complete [11660.268910] LustreError: 407244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.301227] LustreError: 407244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11660.674351] LustreError: 407224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803809 with bad export cookie 13954752032601373190 [11660.678451] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11660.687402] LustreError: 407224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11661.577628] Lustre: server umount lustre-MDT0001 complete [11673.963789] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11676.636858] Key type lgssc unregistered [11676.926687] LNet: 410968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11676.931722] LNetError: 410968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11676.948716] LNet: Removed LNI 192.168.206.138@tcp [11677.834557] Key type .llcrypt unregistered [11677.837241] Key type ._llcrypt unregistered [11705.331910] Key type ._llcrypt registered [11705.336152] Key type .llcrypt registered [11705.554459] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11706.026438] Key type .llcrypt unregistered [11706.033468] Key type ._llcrypt unregistered [11724.279482] Key type ._llcrypt registered [11724.284946] Key type .llcrypt registered [11724.447631] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11725.678362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11725.715285] alg: No test for adler32 (adler32-zlib) [11726.848267] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [11727.029594] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11728.695207] Key type lgssc registered [11729.887589] Lustre: Echo OBD driver; http://www.lustre.org/ [11737.821859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11737.883807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.466325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11744.470846] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11748.848379] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11748.884286] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11753.923494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.953731] LustreError: 413175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.330660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.589327] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.636860] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11762.859903] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11764.707294] LustreError: 413175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.732174] LustreError: 413175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11764.740569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11769.836783] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11769.861111] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11771.942267] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11779.756299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.183083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11785.660265] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11800.545675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11800.548607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.558302] LustreError: Skipped 1 previous similar message [11800.560870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.578847] Lustre: Skipped 1 previous similar message [11800.602417] Lustre: Skipped 1 previous similar message [11802.565858] Lustre: server umount lustre-OST0000 complete [11809.100407] Lustre: server umount lustre-MDT0000 complete [11810.786374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11810.787444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11810.799749] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.799759] LustreError: 413174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11813.282106] LustreError: 413154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803962 with bad export cookie 2218463854161041314 [11813.304118] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.308614] LustreError: 413154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11814.083415] Lustre: server umount lustre-MDT0001 complete [11832.058863] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11836.223370] Key type lgssc unregistered [11836.724410] LNet: 416345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11836.735831] LNetError: 416345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11836.774118] LNet: Removed LNI 192.168.206.138@tcp [11838.718996] Key type .llcrypt unregistered [11838.723053] Key type ._llcrypt unregistered [11864.547437] Key type ._llcrypt registered [11864.554111] Key type .llcrypt registered [11864.660955] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11865.773240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11865.814092] alg: No test for adler32 (adler32-zlib) [11866.913327] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [11867.121868] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11868.791198] Key type lgssc registered [11870.019013] Lustre: Echo OBD driver; http://www.lustre.org/ [11878.395406] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11878.417734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11893.983411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11899.104046] LustreError: 417732:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [11925.728648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11932.086963] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11941.974519] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11950.606718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.179943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11951.323190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11956.195569] LustreError: 417757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11956.242318] LustreError: 417757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11957.091733] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11960.301793] LustreError: 417758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11960.350768] LustreError: 417758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11965.428920] LustreError: 418770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11965.446387] LustreError: 418770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11968.358425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11969.137809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11975.027487] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11982.608562] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11988.882392] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11997.155759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11997.168640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.185025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11999.714553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11999.738905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.770139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12003.064766] Lustre: server umount lustre-OST0000 complete [12009.968102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12009.974277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.994433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.002482] Lustre: Skipped 2 previous similar messages [12010.283318] Lustre: server umount lustre-MDT0000 complete [12010.470862] LustreError: 418773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.483691] LustreError: 418773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12015.287401] LustreError: 417738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804164 with bad export cookie 18251448802037039300 [12015.296468] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12015.890051] Lustre: server umount lustre-MDT0001 complete [12030.266243] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12034.128339] Key type lgssc unregistered [12034.524834] LNet: 421301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12034.537795] LNetError: 421301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12034.554516] LNet: Removed LNI 192.168.206.138@tcp [12035.417065] Key type .llcrypt unregistered [12035.424746] Key type ._llcrypt unregistered [12062.338868] Key type ._llcrypt registered [12062.340996] Key type .llcrypt registered [12062.431379] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12063.510752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12063.711580] alg: No test for adler32 (adler32-zlib) [12064.737505] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [12065.006526] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12066.695246] Key type lgssc registered [12068.570158] Lustre: Echo OBD driver; http://www.lustre.org/ [12077.583802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12077.634771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12079.687263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12084.856830] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12089.829076] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.860363] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12094.932428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.952151] LustreError: 422715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.436851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12100.288225] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12107.618768] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12112.838614] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12120.309024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12120.788033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12124.856375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12128.294346] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12139.184816] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12145.640535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.645298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12145.660776] Lustre: Skipped 2 previous similar messages [12145.681377] Lustre: Skipped 2 previous similar messages [12146.810056] LustreError: 422709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.881390] Lustre: server umount lustre-MDT0000 complete [12149.488504] LustreError: 422694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.38@tcp arrived at 1787804298 with bad export cookie 5313835267313421399 [12149.508219] LustreError: 422694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12150.762315] LustreError: 424304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12150.783713] LustreError: 424304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12151.780043] LustreError: 422696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804300 with bad export cookie 5313835267313420510 [12151.797487] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12151.815672] LustreError: 422696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12155.875973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12155.878375] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.879307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.880093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12155.959653] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12156.908537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12158.123455] Lustre: server umount lustre-OST0000 complete [12164.917631] Lustre: server umount lustre-MDT0001 complete [12175.194994] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12178.146744] Key type lgssc unregistered [12178.523942] LNet: 426397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12178.530825] LNetError: 426397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12178.559988] LNet: Removed LNI 192.168.206.138@tcp [12179.435334] Key type .llcrypt unregistered [12179.444621] Key type ._llcrypt unregistered [12205.291298] Key type ._llcrypt registered [12205.293105] Key type .llcrypt registered [12205.418449] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12207.050901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12207.083603] alg: No test for adler32 (adler32-zlib) [12208.170505] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [12208.520625] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12210.175316] Key type lgssc registered [12211.445845] Lustre: Echo OBD driver; http://www.lustre.org/ [12219.422724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12219.455812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12220.956688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12225.227867] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12231.142701] LustreError: 427814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.162550] LustreError: 427814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12233.560569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12233.847375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12238.171823] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12245.365719] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12250.270167] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12257.954590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12258.485395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12265.847571] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12266.702172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12273.551609] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12280.291490] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12280.297479] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12280.306730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12281.435698] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [12282.340769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12284.293864] Lustre: server umount lustre-MDT0001 complete [12287.457342] LustreError: 427813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.470687] LustreError: 427813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12292.578849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12292.579574] LustreError: 427814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12292.588441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12292.594899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12292.630497] Lustre: Skipped 2 previous similar messages [12295.665774] Lustre: server umount lustre-OST0000 complete [12297.700137] LustreError: 427813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.818767] Lustre: server umount lustre-MDT0000 complete [12309.985111] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12312.142228] Key type lgssc unregistered [12312.370378] LNet: 431495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12312.379148] LNetError: 431495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12312.392799] LNet: Removed LNI 192.168.206.138@tcp [12313.110116] Key type .llcrypt unregistered [12313.114603] Key type ._llcrypt unregistered [12338.940271] Key type ._llcrypt registered [12338.942757] Key type .llcrypt registered [12339.039986] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [12350.870160] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12351.393955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.426143] alg: No test for adler32 (adler32-zlib) [12352.425494] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [12352.604859] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12354.242092] Key type lgssc registered [12355.158686] Lustre: Echo OBD driver; http://www.lustre.org/ [12382.955755] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12394.484979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12394.504839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12395.691503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12395.720278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12395.801609] Lustre: lustre-MDT0000: new disk, initializing [12395.879057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12395.890610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12398.912100] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12409.411371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12409.580773] Lustre: lustre-OST0000: new disk, initializing [12409.584594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12409.587660] Lustre: Skipped 1 previous similar message [12409.591091] Lustre: 436502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12409.640367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12413.418038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12413.425570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12413.435490] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12413.445699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12419.800509] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12431.129882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.239164] Lustre: 435485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12431.305639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12431.364503] Lustre: lustre-MDT0001: new disk, initializing [12431.466714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12431.499966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12431.508211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12434.868483] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12441.271836] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12441.441958] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12444.211450] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12444.347531] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12447.714972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12447.718692] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12447.725800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12447.735993] Lustre: Skipped 1 previous similar message [12448.365410] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [12452.835886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12452.840886] Lustre: Skipped 1 previous similar message [12453.556464] Lustre: server umount lustre-MDT0001 complete [12463.078059] LustreError: 439073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12463.093579] LustreError: 439073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12468.195487] LustreError: 435492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.206065] LustreError: 435492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12469.729194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12469.733906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12469.741442] Lustre: Skipped 1 previous similar message [12469.745371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12469.750337] Lustre: Skipped 1 previous similar message [12473.315701] LustreError: 435497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.703240] Lustre: server umount lustre-OST0000 complete [12483.775707] Lustre: server umount lustre-MDT0000 complete [12493.221909] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12495.039176] Key type lgssc unregistered [12495.264723] LNet: 440247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12495.281382] LNetError: 440247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12495.300583] LNet: Removed LNI 192.168.206.138@tcp [12495.917646] Key type .llcrypt unregistered [12495.920840] Key type ._llcrypt unregistered [12523.267228] Key type ._llcrypt registered [12523.268541] Key type .llcrypt registered [12523.359374] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12523.552468] Key type .llcrypt unregistered [12523.554307] Key type ._llcrypt unregistered [12533.843695] Key type ._llcrypt registered [12533.845389] Key type .llcrypt registered [12533.913790] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12534.571773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12534.633644] alg: No test for adler32 (adler32-zlib) [12535.586191] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [12535.751830] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12537.399218] Key type lgssc registered [12538.191672] Lustre: Echo OBD driver; http://www.lustre.org/ [12557.187586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12557.210635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.418559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12558.436901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12558.490252] Lustre: lustre-MDT0000: new disk, initializing [12558.551734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12558.567684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12561.639654] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12570.108438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.186589] Lustre: 443594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12570.234829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12570.240987] Lustre: Skipped 1 previous similar message [12570.297929] Lustre: lustre-MDT0001: new disk, initializing [12570.376055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12570.419507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12570.438495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12574.020358] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12581.775600] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12586.470901] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12591.966195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12592.188379] Lustre: lustre-OST0000: new disk, initializing [12592.191527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12592.196241] Lustre: 445563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12592.260087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12597.579190] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12600.897694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12600.917403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12601.020345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12605.394531] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12611.565874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12611.571555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.574470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12611.603444] Lustre: Skipped 2 previous similar messages [12616.676359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12616.693503] Lustre: Skipped 4 previous similar messages [12617.120432] Lustre: server umount lustre-MDT0000 complete [12620.121633] LustreError: 443586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804769 with bad export cookie 13126675206358547635 [12620.125653] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12620.131397] LustreError: 443586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12620.387820] Lustre: server umount lustre-MDT0001 complete [12634.258393] Lustre: server umount lustre-OST0000 complete [12643.518266] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:26:31 (1787804791) [12650.518461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12650.960925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12654.972844] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12656.110286] LustreError: 447589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12656.141079] LustreError: 447589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12661.216354] LustreError: 447590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.449769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12662.871207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12666.977878] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12673.634716] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12678.590427] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12697.796186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12698.043406] Lustre: Found index 0 for lustre-OST0000, updating log [12698.185776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12704.000818] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12712.286574] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12722.827744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12728.295747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.296591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12728.328045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12734.332831] Lustre: server umount lustre-OST0000 complete [12750.817680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12750.830708] Lustre: Skipped 1 previous similar message [12750.834634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12750.839380] Lustre: Skipped 4 previous similar messages [12754.979607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12754.982519] Lustre: Skipped 1 previous similar message [12756.841235] Lustre: server umount lustre-MDT0000 complete [12760.035302] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [12760.097309] LustreError: 449429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804909 with bad export cookie 13126675206358549035 [12760.098235] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12760.113376] LustreError: 449429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12760.527624] Lustre: server umount lustre-MDT0001 complete [12769.457668] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:28:36 (1787804916) [13375.454419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13375.741290] LustreError: 454015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13375.803555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13378.843520] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13381.091753] LustreError: 454016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.929203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13385.197449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13388.225431] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13394.949783] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13399.921507] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13418.873414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13419.127639] Lustre: Found index 0 for lustre-OST0000, updating log [13419.418738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13424.565124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13424.774176] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13425.632718] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [13431.139731] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13433.839103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13433.852964] Lustre: Skipped 1 previous similar message [13433.857955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13433.860223] Lustre: Skipped 2 previous similar messages [13447.135356] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13447.295194] Lustre: server umount lustre-OST0000 complete [13451.619027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13451.620413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13451.630426] Lustre: Skipped 1 previous similar message [13453.730992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13453.735511] Lustre: Skipped 3 previous similar messages [13456.802311] LustreError: 454016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13456.841238] Lustre: server umount lustre-MDT0000 complete [13458.856817] LustreError: 454015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13458.885084] LustreError: 454015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13459.648404] LustreError: 455867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805608 with bad export cookie 13126675206358550120 [13459.651237] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13459.657629] LustreError: 455867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13459.868908] Lustre: server umount lustre-MDT0001 complete [13484.533409] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [13485.023582] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [13491.598195] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [13519.436716] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:41:06 (1787805666) [13535.377994] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [13537.597674] Key type lgssc unregistered [13537.804266] LNet: 460622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13537.812043] LNetError: 460622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13637.157883] LNet: Removed LNI 192.168.206.138@tcp [13637.836397] Key type .llcrypt unregistered [13637.840144] Key type ._llcrypt unregistered [13652.107377] Key type ._llcrypt registered [13652.111199] Key type .llcrypt registered [13652.196540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [13653.453960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13653.495041] alg: No test for adler32 (adler32-zlib) [13654.555369] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [13654.894196] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [13656.631160] Key type lgssc registered [13657.482652] Lustre: Echo OBD driver; http://www.lustre.org/ [13662.024513] LNet: Added LNI 192.168.206.138@tcp100 [8/256/0/180] [13663.105902] LNet: Added LNI 192.168.206.138@tcp101 [8/256/0/180] [13664.012928] LNet: Added LNI 192.168.206.138@tcp102 [8/256/0/180] [13664.726569] LNet: Added LNI 192.168.206.138@tcp103 [8/256/0/180] [13665.441876] LNet: Added LNI 192.168.206.138@tcp104 [8/256/0/180] [13666.202087] LNet: Added LNI 192.168.206.138@tcp105 [8/256/0/180] [13666.935420] LNet: Added LNI 192.168.206.138@tcp106 [8/256/0/180] [13667.621763] LNet: Added LNI 192.168.206.138@tcp107 [8/256/0/180] [13668.230342] LNet: Added LNI 192.168.206.138@tcp108 [8/256/0/180] [13669.107397] LNet: Added LNI 192.168.206.138@tcp109 [8/256/0/180] [13670.067933] LNet: Added LNI 192.168.206.138@tcp110 [8/256/0/180] [13671.065046] LNet: Added LNI 192.168.206.138@tcp111 [8/256/0/180] [13671.961508] LNet: Added LNI 192.168.206.138@tcp112 [8/256/0/180] [13672.802922] LNet: Added LNI 192.168.206.138@tcp113 [8/256/0/180] [13673.658693] LNet: Added LNI 192.168.206.138@tcp114 [8/256/0/180] [13674.601568] LNet: Added LNI 192.168.206.138@tcp115 [8/256/0/180] [13675.354630] LNet: Added LNI 192.168.206.138@tcp116 [8/256/0/180] [13676.303476] LNet: Added LNI 192.168.206.138@tcp117 [8/256/0/180] [13677.189749] LNet: Added LNI 192.168.206.138@tcp118 [8/256/0/180] [13677.976337] LNet: Added LNI 192.168.206.138@tcp119 [8/256/0/180] [13678.770606] LNet: Added LNI 192.168.206.138@tcp120 [8/256/0/180] [13679.482849] LNet: Added LNI 192.168.206.138@tcp121 [8/256/0/180] [13680.153326] LNet: Added LNI 192.168.206.138@tcp122 [8/256/0/180] [13680.828910] LNet: Added LNI 192.168.206.138@tcp123 [8/256/0/180] [13681.604408] LNet: Added LNI 192.168.206.138@tcp124 [8/256/0/180] [13682.287472] LNet: Added LNI 192.168.206.138@tcp125 [8/256/0/180] [13682.976176] LNet: Added LNI 192.168.206.138@tcp126 [8/256/0/180] [13683.752390] LNet: Added LNI 192.168.206.138@tcp127 [8/256/0/180] [13684.432676] LNet: Added LNI 192.168.206.138@tcp128 [8/256/0/180] [13685.082723] LNet: Added LNI 192.168.206.138@tcp129 [8/256/0/180] [13685.756696] LNet: Added LNI 192.168.206.138@tcp130 [8/256/0/180] [13686.442320] LNet: Added LNI 192.168.206.138@tcp131 [8/256/0/180] [13687.153480] LNet: Added LNI 192.168.206.138@tcp132 [8/256/0/180] [13687.853340] LNet: Added LNI 192.168.206.138@tcp133 [8/256/0/180] [13688.490609] LNet: Added LNI 192.168.206.138@tcp134 [8/256/0/180] [13689.175499] LNet: Added LNI 192.168.206.138@tcp135 [8/256/0/180] [13694.030696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13694.043377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13695.135966] Lustre: 463627:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [13695.244908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13695.280740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13695.368203] Lustre: lustre-MDT0000: new disk, initializing [13695.460205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13695.475953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13698.713547] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13707.442116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13707.586228] Lustre: 463645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13707.679713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13707.686489] Lustre: Skipped 1 previous similar message [13707.794754] Lustre: lustre-MDT0001: new disk, initializing [13707.846733] Lustre: 464441:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [13707.857530] Lustre: 464441:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13708.030432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13708.065949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13708.081831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13710.687085] Lustre: 464604:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [13710.710682] Lustre: 464604:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13712.501415] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13722.584305] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13728.684651] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13735.584516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13735.887345] Lustre: lustre-OST0000: new disk, initializing [13735.890959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13735.900407] Lustre: 465614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13735.974477] Lustre: 465619:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [13735.981494] Lustre: 465619:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13736.032482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13741.606759] Lustre: 465963:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [13741.620535] Lustre: 465963:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13741.653950] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13741.684370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13741.696255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13741.845707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13750.229389] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13757.418475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13757.418515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13757.436187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13757.440993] LustreError: Skipped 1 previous similar message [13757.446073] Lustre: Skipped 1 previous similar message [13762.494828] Lustre: server umount lustre-OST0000 complete [13767.650331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13767.654670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13767.662457] Lustre: Skipped 2 previous similar messages [13771.905280] Lustre: server umount lustre-MDT0000 complete [13775.384390] LustreError: 463636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805924 with bad export cookie 5712869341798042536 [13775.396166] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13775.417681] LustreError: 463636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13775.686482] Lustre: server umount lustre-MDT0001 complete [13786.019869] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [13788.275817] Key type lgssc unregistered [13788.565603] LNet: 467523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13788.573404] LNetError: 467523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13788.583101] LNet: Removed LNI 192.168.206.138@tcp [13789.599180] LNet: Removed LNI 192.168.206.138@tcp100 [13789.602311] LNetError: 467523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13789.607709] LNetError: 467523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13789.615700] LNet: Removed LNI 192.168.206.138@tcp101 [13789.622018] LNet: Removed LNI 192.168.206.138@tcp102 [13789.627181] LNet: Removed LNI 192.168.206.138@tcp103 [13789.633928] LNet: Removed LNI 192.168.206.138@tcp104 [13789.638661] LNet: Removed LNI 192.168.206.138@tcp105 [13789.642116] LNet: Removed LNI 192.168.206.138@tcp106 [13789.653286] LNet: Removed LNI 192.168.206.138@tcp107 [13789.656580] LNet: Removed LNI 192.168.206.138@tcp108 [13789.666191] LNet: Removed LNI 192.168.206.138@tcp109 [13789.669474] LNet: Removed LNI 192.168.206.138@tcp110 [13789.672182] LNet: Removed LNI 192.168.206.138@tcp111 [13789.675535] LNet: Removed LNI 192.168.206.138@tcp112 [13789.678398] LNet: Removed LNI 192.168.206.138@tcp113 [13789.683339] LNet: Removed LNI 192.168.206.138@tcp114 [13789.686179] LNet: Removed LNI 192.168.206.138@tcp115 [13789.689135] LNet: Removed LNI 192.168.206.138@tcp116 [13789.692176] LNet: Removed LNI 192.168.206.138@tcp117 [13789.695407] LNet: Removed LNI 192.168.206.138@tcp118 [13789.702151] LNet: Removed LNI 192.168.206.138@tcp119 [13789.707674] LNet: Removed LNI 192.168.206.138@tcp120 [13789.711301] LNet: Removed LNI 192.168.206.138@tcp121 [13789.714475] LNet: Removed LNI 192.168.206.138@tcp122 [13789.717284] LNet: Removed LNI 192.168.206.138@tcp123 [13789.721111] LNet: Removed LNI 192.168.206.138@tcp124 [13789.724179] LNet: Removed LNI 192.168.206.138@tcp125 [13789.728265] LNet: Removed LNI 192.168.206.138@tcp126 [13789.731174] LNet: Removed LNI 192.168.206.138@tcp127 [13789.739720] LNet: Removed LNI 192.168.206.138@tcp128 [13789.741746] LNet: Removed LNI 192.168.206.138@tcp129 [13789.751241] LNet: Removed LNI 192.168.206.138@tcp130 [13789.758857] LNet: Removed LNI 192.168.206.138@tcp131 [13789.766684] LNet: Removed LNI 192.168.206.138@tcp132 [13789.771631] LNet: Removed LNI 192.168.206.138@tcp133 [13789.777105] LNet: Removed LNI 192.168.206.138@tcp134 [13789.791757] LNet: Removed LNI 192.168.206.138@tcp135 [13790.438754] Key type .llcrypt unregistered [13790.441534] Key type ._llcrypt unregistered [13807.999595] Key type ._llcrypt registered [13808.003060] Key type .llcrypt registered [13808.103805] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [13818.993065] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [13819.713231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13819.732907] alg: No test for adler32 (adler32-zlib) [13820.763330] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [13820.930260] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [13822.583193] Key type lgssc registered [13823.344391] Lustre: Echo OBD driver; http://www.lustre.org/ [13859.838639] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:46:46 (1787806006) [13877.065159] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [13879.510218] Key type lgssc unregistered [13879.748664] LNet: 471581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13879.758168] LNetError: 471581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13879.777806] LNet: Removed LNI 192.168.206.138@tcp [13880.720409] Key type .llcrypt unregistered [13880.726692] Key type ._llcrypt unregistered [13894.229383] Key type ._llcrypt registered [13894.231102] Key type .llcrypt registered [13894.341668] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [13895.224207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13895.255543] alg: No test for adler32 (adler32-zlib) [13896.347787] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [13896.638747] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [13898.335216] Key type lgssc registered [13899.315349] Lustre: Echo OBD driver; http://www.lustre.org/ [13909.196993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13909.240293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13910.482274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13910.518908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13910.651814] Lustre: lustre-MDT0000: new disk, initializing [13910.783073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13910.805182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13914.898022] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13925.838500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13925.988074] Lustre: 472770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13926.142616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13926.146748] Lustre: Skipped 1 previous similar message [13926.206521] Lustre: lustre-MDT0001: new disk, initializing [13926.281262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13926.306125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13926.327950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13931.410242] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13941.462191] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13946.738515] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13952.730427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13952.998359] Lustre: lustre-OST0000: new disk, initializing [13953.002814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13953.009572] Lustre: 474739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13953.095463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13958.063769] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13961.760276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13961.765804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13961.838429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13966.312172] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13969.486363] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lnet_if_list [13974.331854] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13974.380623] LNet: Added LNI 192.168.206.138@tcp10 [8/256/0/180] [13980.240118] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13980.259990] LNet: Added LNI 192.168.206.138@tcp11 [8/256/0/180] [13984.827478] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13984.850618] LNet: Added LNI 192.168.206.138@tcp12 [8/256/0/180] [13990.217915] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13990.250923] LNet: Added LNI 192.168.206.138@tcp13 [8/256/0/180] [13995.900510] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13995.962950] LNet: Added LNI 192.168.206.138@tcp14 [8/256/0/180] [14001.197066] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14001.225741] LNet: Added LNI 192.168.206.138@tcp15 [8/256/0/180] [14004.849092] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14004.892515] LNet: Added LNI 192.168.206.138@tcp16 [8/256/0/180] [14009.762687] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14009.792977] LNet: Added LNI 192.168.206.138@tcp17 [8/256/0/180] [14014.121466] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14014.166543] LNet: Added LNI 192.168.206.138@tcp18 [8/256/0/180] [14018.302763] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14018.321384] LNet: Added LNI 192.168.206.138@tcp19 [8/256/0/180] [14022.305800] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14022.344645] LNet: Added LNI 192.168.206.138@tcp20 [8/256/0/180] [14027.793346] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14027.822995] LNet: Added LNI 192.168.206.138@tcp21 [8/256/0/180] [14032.775668] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14032.793035] LNet: Added LNI 192.168.206.138@tcp22 [8/256/0/180] [14038.305745] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14038.326888] LNet: Added LNI 192.168.206.138@tcp23 [8/256/0/180] [14043.817468] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14043.867244] LNet: Added LNI 192.168.206.138@tcp24 [8/256/0/180] [14051.975741] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14052.023717] LNet: Added LNI 192.168.206.138@tcp25 [8/256/0/180] [14060.542465] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14060.567537] LNet: Added LNI 192.168.206.138@tcp26 [8/256/0/180] [14067.892870] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14067.927557] LNet: Added LNI 192.168.206.138@tcp27 [8/256/0/180] [14076.176907] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14076.229561] LNet: Added LNI 192.168.206.138@tcp28 [8/256/0/180] [14085.253645] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14085.285589] LNet: Added LNI 192.168.206.138@tcp29 [8/256/0/180] [14092.090335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14092.138274] LNet: Added LNI 192.168.206.138@tcp30 [8/256/0/180] [14098.517884] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14098.563521] LNet: Added LNI 192.168.206.138@tcp31 [8/256/0/180] [14105.781085] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14105.820794] LNet: Added LNI 192.168.206.138@tcp32 [8/256/0/180] [14115.133192] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14115.185871] LNet: Added LNI 192.168.206.138@tcp33 [8/256/0/180] [14123.927423] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14123.972871] LNet: Added LNI 192.168.206.138@tcp34 [8/256/0/180] [14131.186121] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14131.216381] LNet: Added LNI 192.168.206.138@tcp35 [8/256/0/180] [14139.111811] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14139.139057] LNet: Added LNI 192.168.206.138@tcp36 [8/256/0/180] [14148.318284] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14148.386104] LNet: Added LNI 192.168.206.138@tcp37 [8/256/0/180] [14156.007434] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14156.050602] LNet: Added LNI 192.168.206.138@tcp38 [8/256/0/180] [14163.594602] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14163.616796] LNet: Added LNI 192.168.206.138@tcp39 [8/256/0/180] [14170.560173] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14170.583820] LNet: Added LNI 192.168.206.138@tcp40 [8/256/0/180] [14177.758441] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14177.786579] LNet: Added LNI 192.168.206.138@tcp41 [8/256/0/180] [14185.639604] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14185.682340] LNet: Added LNI 192.168.206.138@tcp42 [8/256/0/180] [14194.491838] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14194.541235] LNet: Added LNI 192.168.206.138@tcp43 [8/256/0/180] [14203.563169] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14203.583443] LNet: Added LNI 192.168.206.138@tcp44 [8/256/0/180] [14211.119354] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14211.154249] LNet: Added LNI 192.168.206.138@tcp45 [8/256/0/180] [14217.795639] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14217.823720] LNet: Added LNI 192.168.206.138@tcp46 [8/256/0/180] [14224.693945] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14224.727665] LNet: Added LNI 192.168.206.138@tcp47 [8/256/0/180] [14233.876481] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14233.912486] LNet: Added LNI 192.168.206.138@tcp48 [8/256/0/180] [14245.053751] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14245.075971] LNet: Added LNI 192.168.206.138@tcp49 [8/256/0/180] [14253.209424] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14253.248188] LNet: Added LNI 192.168.206.138@tcp50 [8/256/0/180] [14274.018263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14274.023482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14274.045031] Lustre: Skipped 2 previous similar messages [14274.058251] Lustre: Skipped 2 previous similar messages [14278.495260] Lustre: server umount lustre-MDT0000 complete [14281.870140] LustreError: 472763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806431 with bad export cookie 293156925128574068 [14281.885653] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14281.888227] LustreError: 472763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14282.270773] Lustre: server umount lustre-MDT0001 complete [14299.863114] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14300.639644] Lustre: 472042:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806433/real 1787806433] req@000000005f676643 x1874650511619200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14300.662424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14302.751214] Lustre: 472042:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806435/real 1787806435] req@000000003c6e65b9 x1874650511619328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14305.759304] Lustre: 472040:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806438/real 1787806438] req@000000003d09175f x1874650511619456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14306.532860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14307.028304] LustreError: 474732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14307.042392] LustreError: 474732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14307.077914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14308.128693] Lustre: 472041:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806441/real 1787806441] req@0000000061e1f480 x1874650511619712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14309.094978] LustreError: 474733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14310.111577] LustreError: 474732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14310.146101] LustreError: 474732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14311.084328] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14313.441125] Lustre: 472043:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787806446/real 1787806446] req@00000000ed6ffd8b x1874650511619840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787806462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14315.491426] LustreError: 498979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14315.501891] LustreError: 498979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14318.466168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14318.764255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14322.298833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14324.202372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14328.732963] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14329.334701] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14334.340069] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14353.377085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14353.394749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14353.412941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14354.918871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14354.926218] Lustre: Skipped 1 previous similar message [14360.037180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14365.153458] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14365.163338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14365.180516] Lustre: Skipped 2 previous similar messages [14365.429697] Lustre: server umount lustre-OST0000 complete [14375.397360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14375.397451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14375.435178] Lustre: Skipped 2 previous similar messages [14375.436292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14375.443200] Lustre: Skipped 1 previous similar message [14377.038473] Lustre: server umount lustre-MDT0000 complete [14380.514683] LustreError: 498984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14380.532657] LustreError: 498984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14381.518279] LustreError: 472762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806530 with bad export cookie 293156925128576728 [14381.521398] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14381.525080] LustreError: 472762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14381.869124] Lustre: server umount lustre-MDT0001 complete [14392.411155] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14394.802523] Key type lgssc unregistered [14395.170722] LNet: 501759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14395.182289] LNetError: 501759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14395.191393] LNet: Removed LNI 192.168.206.138@tcp [14395.193890] LNet: Removed LNI 192.168.206.138@tcp10 [14395.196911] LNet: Removed LNI 192.168.206.138@tcp11 [14395.202417] LNet: Removed LNI 192.168.206.138@tcp12 [14395.206469] LNet: Removed LNI 192.168.206.138@tcp13 [14395.210626] LNet: Removed LNI 192.168.206.138@tcp14 [14395.216623] LNet: Removed LNI 192.168.206.138@tcp15 [14395.220764] LNet: Removed LNI 192.168.206.138@tcp16 [14395.223902] LNet: Removed LNI 192.168.206.138@tcp17 [14395.227767] LNet: Removed LNI 192.168.206.138@tcp18 [14395.229647] LNet: Removed LNI 192.168.206.138@tcp19 [14395.232816] LNet: Removed LNI 192.168.206.138@tcp20 [14395.240148] LNet: Removed LNI 192.168.206.138@tcp21 [14395.243176] LNet: Removed LNI 192.168.206.138@tcp22 [14395.245848] LNet: Removed LNI 192.168.206.138@tcp23 [14395.249160] LNet: Removed LNI 192.168.206.138@tcp24 [14395.254158] LNet: Removed LNI 192.168.206.138@tcp25 [14395.258118] LNet: Removed LNI 192.168.206.138@tcp26 [14395.261689] LNet: Removed LNI 192.168.206.138@tcp27 [14395.265869] LNet: Removed LNI 192.168.206.138@tcp28 [14395.268798] LNet: Removed LNI 192.168.206.138@tcp29 [14395.271779] LNet: Removed LNI 192.168.206.138@tcp30 [14395.278802] LNet: Removed LNI 192.168.206.138@tcp31 [14395.286865] LNet: Removed LNI 192.168.206.138@tcp32 [14395.289843] LNet: Removed LNI 192.168.206.138@tcp33 [14395.294524] LNet: Removed LNI 192.168.206.138@tcp34 [14395.296676] LNet: Removed LNI 192.168.206.138@tcp35 [14395.299883] LNet: Removed LNI 192.168.206.138@tcp36 [14395.302804] LNet: Removed LNI 192.168.206.138@tcp37 [14395.305272] LNet: Removed LNI 192.168.206.138@tcp38 [14395.308248] LNet: Removed LNI 192.168.206.138@tcp39 [14395.311610] LNet: Removed LNI 192.168.206.138@tcp40 [14395.314568] LNet: Removed LNI 192.168.206.138@tcp41 [14395.318254] LNet: Removed LNI 192.168.206.138@tcp42 [14395.321623] LNet: Removed LNI 192.168.206.138@tcp43 [14395.325582] LNet: Removed LNI 192.168.206.138@tcp44 [14395.333120] LNet: Removed LNI 192.168.206.138@tcp45 [14395.336793] LNet: Removed LNI 192.168.206.138@tcp46 [14395.342598] LNet: Removed LNI 192.168.206.138@tcp47 [14395.350586] LNet: Removed LNI 192.168.206.138@tcp48 [14395.355898] LNet: Removed LNI 192.168.206.138@tcp49 [14395.385089] LNet: Removed LNI 192.168.206.138@tcp50 [14396.141357] Key type .llcrypt unregistered [14396.147324] Key type ._llcrypt unregistered [14417.805419] Key type ._llcrypt registered [14417.807828] Key type .llcrypt registered [14417.984071] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [14439.880836] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14441.398808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14441.428597] alg: No test for adler32 (adler32-zlib) [14442.840072] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [14443.260153] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14444.959268] Key type lgssc registered [14447.312869] Lustre: Echo OBD driver; http://www.lustre.org/ [14486.285487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14486.317290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14487.546184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14487.599732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14487.684051] Lustre: lustre-MDT0000: new disk, initializing [14487.736080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14487.751270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14492.330745] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14504.423081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14504.661454] Lustre: 504954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14504.798854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14504.813208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14504.815045] Lustre: Skipped 1 previous similar message [14504.956635] Lustre: lustre-MDT0001: new disk, initializing [14505.111144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14505.147127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14505.182264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14511.175116] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14521.131575] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14527.816918] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14534.483089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14534.683537] Lustre: *** cfs_fail_loc=90f, val=0*** [14534.810269] Lustre: lustre-OST0000: new disk, initializing [14534.813983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14534.821704] Lustre: 506970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14535.027025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14540.747889] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14541.359716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14541.365871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14541.485815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14550.857393] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14562.290289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14562.299309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14562.309572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14562.324518] Lustre: Skipped 1 previous similar message [14562.358204] Lustre: Skipped 1 previous similar message [14566.845086] Lustre: server umount lustre-OST0000 complete [14571.492756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14571.502575] LustreError: Skipped 1 previous similar message [14571.508841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14571.520435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14572.514889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14577.633128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14577.641739] Lustre: Skipped 3 previous similar messages [14582.754879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14582.765782] Lustre: Skipped 1 previous similar message [14585.311122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14585.520095] Lustre: server umount lustre-MDT0000 complete [14589.255874] LustreError: 504948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806738 with bad export cookie 2516934310914012331 [14589.264269] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14589.274853] LustreError: 504948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14589.817724] Lustre: server umount lustre-MDT0001 complete [14602.354212] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14606.528807] Key type lgssc unregistered [14606.977771] LNet: 508878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14606.995817] LNetError: 508878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14608.049543] LNet: Removed LNI 192.168.206.138@tcp [14609.393621] Key type .llcrypt unregistered [14609.401833] Key type ._llcrypt unregistered [14629.938713] Key type ._llcrypt registered [14629.944933] Key type .llcrypt registered [14630.103083] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14631.111200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14631.186127] alg: No test for adler32 (adler32-zlib) [14632.171252] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [14632.372421] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14634.023244] Key type lgssc registered [14635.037669] Lustre: Echo OBD driver; http://www.lustre.org/ [14642.769397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14642.783289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14644.338836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14649.192385] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14653.922278] LustreError: 510144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14653.944602] LustreError: 510144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14657.696100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14658.044824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14662.185662] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14669.093468] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14674.256835] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14680.760178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14681.156896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14687.179695] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14693.650347] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14705.044879] loop0: detected capacity change from 0 to 204800000 [14705.264652] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14705.289762] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14705.304175] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14711.277894] loop0: detected capacity change from 0 to 204800000 [14711.504240] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14711.523968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14711.552551] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14717.722406] loop0: detected capacity change from 0 to 204800000 [14725.100449] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14725.262507] Lustre: fs15246-OST0000: new disk, initializing [14725.272836] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14725.280832] Lustre: 513755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14725.375185] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14731.945778] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14742.108517] loop1: detected capacity change from 0 to 204800000 [14746.684736] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14746.757594] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14746.804786] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14746.937895] Lustre: fs15246-MDT0000: new disk, initializing [14747.125696] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14747.156573] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14747.226253] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14748.256502] Lustre: 514877:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000dcaa4929 x1874651283241472/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787806913 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14751.258917] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14757.350282] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14757.373409] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14757.401734] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14782.441779] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14782.481411] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14784.771988] Lustre: server umount fs15246-MDT0000 complete [14787.557562] LustreError: 511885:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14792.693169] LustreError: 510139:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14795.071210] Lustre: server umount fs15246-OST0000 complete [14811.105834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14811.114540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14811.140964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14812.133219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14812.145574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14812.168994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14815.820611] Lustre: server umount lustre-OST0000 complete [14821.857948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14821.864207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14821.879717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14821.883591] Lustre: Skipped 2 previous similar messages [14826.069271] Lustre: server umount lustre-MDT0000 complete [14828.513388] LustreError: 510141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14828.541401] LustreError: 510141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14830.292066] LustreError: 510125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806979 with bad export cookie 6640098833568762874 [14830.295969] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14830.304439] LustreError: 510125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14830.612959] Lustre: server umount lustre-MDT0001 complete [14845.698717] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14849.362341] Key type lgssc unregistered [14849.770284] LNet: 517975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14849.774798] LNetError: 517975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.783975] LNet: Removed LNI 192.168.206.138@tcp [14851.106314] Key type .llcrypt unregistered [14851.109324] Key type ._llcrypt unregistered [14870.326442] Key type ._llcrypt registered [14870.329580] Key type .llcrypt registered [14870.471509] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14872.495405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14872.535646] alg: No test for adler32 (adler32-zlib) [14873.970867] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [14874.415553] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14876.073512] Key type lgssc registered [14877.272843] Lustre: Echo OBD driver; http://www.lustre.org/ [14919.686468] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [14931.509693] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14964.840680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14964.884691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14966.125304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14966.159242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14966.261324] Lustre: lustre-MDT0000: new disk, initializing [14966.348553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14966.368397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14970.402212] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14979.715216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14979.812150] Lustre: 522437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14979.855296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14979.858208] Lustre: Skipped 1 previous similar message [14979.922268] Lustre: lustre-MDT0001: new disk, initializing [14980.018785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14980.065426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14980.086117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14983.695633] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14992.672504] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14997.784322] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15003.884919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15004.190983] Lustre: lustre-OST0000: new disk, initializing [15004.195341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15004.202148] Lustre: 524405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15004.277602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15009.705391] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15012.420276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15012.439395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15012.599562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15018.343904] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15022.941254] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15023.192560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15027.210704] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15027.519789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15033.318278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15033.323685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15033.329951] LustreError: Skipped 1 previous similar message [15033.331916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15033.383937] Lustre: Skipped 1 previous similar message [15035.861951] Lustre: server umount lustre-OST0000 complete [15041.507105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15041.527883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15041.563035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15041.570212] Lustre: Skipped 1 previous similar message [15043.556573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15043.564840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15045.606318] Lustre: server umount lustre-MDT0000 complete [15049.833396] LustreError: 522430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807199 with bad export cookie 4207007742663927806 [15049.840017] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15049.848147] LustreError: 522430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15050.378642] Lustre: server umount lustre-MDT0001 complete [15076.073629] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [15087.418975] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [15119.289721] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14832 sec ========= 01:07:46 (1787807266) [15120.823588] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:07:48 (1787807268) === [15124.469474] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:07:51 (1787807271) === [15160.237833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [15162.534902] Key type lgssc unregistered [15162.777661] LNet: 531046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15162.783791] LNetError: 531046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15163.828743] LNet: Removed LNI 192.168.206.138@tcp [15164.558312] Key type .llcrypt unregistered [15164.560272] Key type ._llcrypt unregistered