[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492640442 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003336] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007724] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011021] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.021803] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026496] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.030418] *** VALIDATE proc *** [ 0.031256] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040297] debug: unmapping init [mem 0xffffffffb7659000-0xffffffffb7660fff] [ 0.042173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043669] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051288] rcu: Hierarchical SRCU implementation. [ 0.053158] smp: Bringing up secondary CPUs ... [ 0.054509] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058430] smp: Brought up 1 node, 4 CPUs [ 0.060017] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128058] node 0 deferred pages initialised in 65ms [ 0.132021] devtmpfs: initialized [ 0.133316] x86/mm: Memory block size: 128MB [ 0.136890] gcov: version magic: 0x41383552 [ 0.140282] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142295] pinctrl core: initialized pinctrl subsystem [ 0.144202] [ 0.144797] ************************************************************* [ 0.148014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150011] ** ** [ 0.153012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.162015] ** IOMMU data structures, which may compromise security on ** [ 0.164010] ** your system. ** [ 0.167015] ** ** [ 0.169010] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.174021] ** ** [ 0.177022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179012] ************************************************************* [ 0.183257] NET: Registered protocol family 16 [ 0.185438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195089] cpuidle: using governor menu [ 0.196668] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199413] PCI: Using configuration type 1 for base access [ 0.202158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213150] cryptd: max_cpu_qlen set to 1000 [ 0.216249] ACPI: Added _OSI(Module Device) [ 0.218014] ACPI: Added _OSI(Processor Device) [ 0.220013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227009] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233565] ACPI: Interpreter enabled [ 0.235070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237101] ACPI: Using IOAPIC for interrupt routing [ 0.239128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254676] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258164] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272857] acpiphp: Slot [2] registered [ 0.274125] acpiphp: Slot [5] registered [ 0.276219] acpiphp: Slot [6] registered [ 0.278173] acpiphp: Slot [7] registered [ 0.280153] acpiphp: Slot [8] registered [ 0.282160] acpiphp: Slot [9] registered [ 0.284115] acpiphp: Slot [10] registered [ 0.286189] acpiphp: Slot [3] registered [ 0.288130] acpiphp: Slot [4] registered [ 0.290116] acpiphp: Slot [11] registered [ 0.292100] acpiphp: Slot [12] registered [ 0.294103] acpiphp: Slot [13] registered [ 0.295106] acpiphp: Slot [14] registered [ 0.297142] acpiphp: Slot [15] registered [ 0.299121] acpiphp: Slot [16] registered [ 0.300126] acpiphp: Slot [17] registered [ 0.302115] acpiphp: Slot [18] registered [ 0.304119] acpiphp: Slot [19] registered [ 0.305115] acpiphp: Slot [20] registered [ 0.307113] acpiphp: Slot [21] registered [ 0.308106] acpiphp: Slot [22] registered [ 0.310125] acpiphp: Slot [23] registered [ 0.312109] acpiphp: Slot [24] registered [ 0.313120] acpiphp: Slot [25] registered [ 0.315099] acpiphp: Slot [26] registered [ 0.317133] acpiphp: Slot [27] registered [ 0.319117] acpiphp: Slot [28] registered [ 0.321118] acpiphp: Slot [29] registered [ 0.323222] acpiphp: Slot [30] registered [ 0.325122] acpiphp: Slot [31] registered [ 0.327066] PCI host bridge to bus 0000:00 [ 0.329024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353919] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358858] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370806] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375028] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388474] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390647] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.393037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.401013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.427958] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.434206] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466118] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520240] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.555020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.566958] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.587000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619816] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.625020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.666556] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.688019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.704018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.721173] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.723537] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726353] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.728410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.730153] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733234] iommu: Default domain type: Passthrough [ 0.735510] SCSI subsystem initialized [ 0.737113] ACPI: bus type USB registered [ 0.738098] usbcore: registered new interface driver usbfs [ 0.741080] usbcore: registered new interface driver hub [ 0.743079] usbcore: registered new device driver usb [ 0.745155] pps_core: LinuxPPS API ver. 1 registered [ 0.747009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.749054] PTP clock support registered [ 0.751130] EDAC MC: Ver: 3.0.0 [ 0.753124] PCI: Using ACPI for IRQ routing [ 0.754910] NetLabel: Initializing [ 0.756011] NetLabel: domain hash size = 128 [ 0.758012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.760080] NetLabel: unlabeled traffic allowed by default [ 0.762264] vgaarb: loaded [ 0.764338] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.773153] clocksource: Switched to clocksource kvm-clock [ 0.882581] VFS: Disk quotas dquot_6.6.0 [ 0.885273] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889144] *** VALIDATE ramfs *** [ 0.890718] *** VALIDATE hugetlbfs *** [ 0.892449] pnp: PnP ACPI init [ 0.895564] pnp: PnP ACPI: found 6 devices [ 0.915069] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.919161] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.922696] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.925556] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.930648] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.933732] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.937055] NET: Registered protocol family 2 [ 0.939929] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945518] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.949417] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.955716] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.959741] TCP: Hash tables configured (established 65536 bind 65536) [ 0.963245] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.967339] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970743] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975359] NET: Registered protocol family 1 [ 0.979504] RPC: Registered named UNIX socket transport module. [ 0.981903] RPC: Registered udp transport module. [ 0.983224] RPC: Registered tcp transport module. [ 0.984506] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986564] NET: Registered protocol family 44 [ 0.987812] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989668] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991350] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993750] PCI: CLS 0 bytes, default 64 [ 0.995717] Unpacking initramfs... [ 3.854178] debug: unmapping init [mem 0xffff9299fcc54000-0xffff9299fffbffff] [ 3.866088] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.872686] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.885227] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.908458] Initialise system trusted keyrings [ 4.910626] Key type blacklist registered [ 4.915751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.953567] zbud: loaded [ 4.969333] *** VALIDATE nfs *** [ 4.971561] *** VALIDATE nfs4 *** [ 4.985078] pstore: using deflate compression [ 5.004675] Platform Keyring initialized [ 5.220847] NET: Registered protocol family 38 [ 5.223512] Key type asymmetric registered [ 5.226297] Asymmetric key parser 'x509' registered [ 5.229397] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.234455] io scheduler mq-deadline registered [ 5.237029] io scheduler kyber registered [ 5.240665] io scheduler bfq registered [ 5.244103] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.248600] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.252261] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.255259] ACPI: Power Button [PWRF] [ 5.262784] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.274631] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.299804] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.313710] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.342848] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.374126] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.404734] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.411181] Non-volatile memory driver v1.3 [ 5.413298] Linux agpgart interface v0.103 [ 5.464818] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.474183] vda: detected capacity change from 0 to 74690560 [ 5.516424] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.523113] vdb: detected capacity change from 0 to 1073741824 [ 5.571502] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.582740] vdc: detected capacity change from 0 to 2621440000 [ 5.633601] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.641555] vdd: detected capacity change from 0 to 2621440000 [ 5.689201] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.697050] vde: detected capacity change from 0 to 4294967296 [ 5.730449] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.740880] vdf: detected capacity change from 0 to 4294967296 [ 5.757120] libphy: Fixed MDIO Bus: probed [ 5.801272] usbcore: registered new interface driver usbserial_generic [ 5.805367] usbserial: USB Serial support registered for generic [ 5.810273] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.816546] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.819158] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.823211] mousedev: PS/2 mouse device common for all mice [ 5.827329] rtc_cmos 00:05: RTC can wake from S4 [ 5.832668] rtc_cmos 00:05: registered as rtc0 [ 5.835678] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.840016] intel_pstate: CPU model not supported [ 5.842720] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.845304] hid: raw HID events driver (C) Jiri Kosina [ 5.857454] usbcore: registered new interface driver usbhid [ 5.868828] usbhid: USB HID core driver [ 5.871041] drop_monitor: Initializing network drop monitor service [ 5.877242] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.881819] Initializing XFRM netlink socket [ 5.893273] NET: Registered protocol family 10 [ 5.899444] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.901807] Segment Routing with IPv6 [ 5.908906] NET: Registered protocol family 17 [ 5.912388] mpls_gso: MPLS GSO support [ 5.931105] RAS: Correctable Errors collector initialized. [ 5.933099] AVX version of gcm_enc/dec engaged. [ 5.947329] AES CTR mode by8 optimization enabled [ 6.179735] sched_clock: Marking stable (6179714224, 0)->(7099283101, -919568877) [ 6.194976] registered taskstats version 1 [ 6.200280] Loading compiled-in X.509 certificates [ 6.204232] zswap: loaded using pool lzo/zbud [ 6.292510] Key type big_key registered [ 6.317224] Key type encrypted registered [ 6.319925] ima: No TPM chip found, activating TPM-bypass! [ 6.323475] ima: Allocated hash algorithm: sha1 [ 6.327476] ima: No architecture policies found [ 6.332290] evm: Initialising EVM extended attributes: [ 6.335732] evm: security.selinux [ 6.343660] evm: security.ima [ 6.349236] evm: security.capability [ 6.356705] evm: HMAC attrs: 0x1 [ 6.362292] rtc_cmos 00:05: setting system clock to 2026-07-25 00:02:15 UTC (1784937735) [ 6.371340] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 6.378901] debug: unmapping init [mem 0xffffffffb7382000-0xffffffffb7658fff] [ 6.392119] Write protecting the kernel read-only data: 28672k [ 6.399619] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 6.404768] debug: unmapping init [mem 0xffffffffb6314000-0xffffffffb63fffff] [ 6.543352] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.557770] systemd[1]: Detected virtualization kvm. [ 6.561668] systemd[1]: Detected architecture x86-64. [ 6.564883] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.700501] systemd[1]: No hostname configured. [ 6.703863] systemd[1]: Set hostname to . [ 6.712838] random: systemd: uninitialized urandom read (16 bytes read) [ 6.724215] systemd[1]: Initializing machine ID from random generator. [ 6.842131] random: ln: uninitialized urandom read (6 bytes read) [ 7.102291] random: systemd: uninitialized urandom read (16 bytes read) [ 7.107110] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.118681] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 7.127409] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.878682] device-mapper: uevent: version 1.0.3 [ 8.884455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.687399] random: fast init done [ 10.865486] virtio_net virtio0 ens2: renamed from eth0 [ 10.982255] scsi host0: ata_piix [ 11.014217] scsi host1: ata_piix [ 11.016088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.019057] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.879614] random: crng init done [ 15.882632] random: 7 urandom warning(s) missed due to ratelimiting [ 18.479467] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.168699] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.212114] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.292436] SELinux: Disabled at runtime. [ 25.373904] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.382483] systemd[1]: Detected virtualization kvm. [ 25.384388] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.914129] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.920218] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.952090] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.964265] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.972756] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.991636] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.015942] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 27.570808] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 28.703544] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.685153] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.770623] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.246651] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.428856] EDAC sbridge: Ver: 1.1.2 [ 33.665886] Key type dns_resolver registered [ 34.267052] NFS: Registering the id_resolver key type [ 34.280876] Key type id_resolver registered [ 34.290130] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-server login: [ 47.901636] hrtimer: interrupt took 7627107 ns [ 90.718790] libcfs: loading out-of-tree module taints kernel. [ 90.796989] Key type ._llcrypt registered [ 90.800691] Key type .llcrypt registered [ 90.895205] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 111.809802] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 113.533860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.553568] alg: No test for adler32 (adler32-zlib) [ 115.368335] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 117.308329] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 119.351323] Key type lgssc registered [ 122.476452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.755625] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.808158] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 202.298962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.328428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.614663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.651427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.755476] Lustre: lustre-MDT0000: new disk, initializing [ 203.864457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.876674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.558723] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.751196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.906773] Lustre: 6529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.952344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.957380] Lustre: Skipped 1 previous similar message [ 223.041429] Lustre: lustre-MDT0001: new disk, initializing [ 223.104591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.141437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.166539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.219420] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.363244] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.878604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.175238] Lustre: lustre-OST0000: new disk, initializing [ 241.182872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.186444] Lustre: 8467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.247828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.317425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.342251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.436427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.480299] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.533886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.691588] Lustre: lustre-OST0001: new disk, initializing [ 260.701727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.714034] Lustre: 9538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.790763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.556531] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.376052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.388838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.486099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.591146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.091783] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.412679] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.008457] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing yml_node [ 298.554966] Lustre: DEBUG MARKER: Client: 2.17.54.203 [ 300.709972] Lustre: DEBUG MARKER: MDS: 2.17.54.203 [ 303.432677] Lustre: DEBUG MARKER: OSS: 2.17.54.203 [ 304.958336] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 20:07:12 EDT 2026 [ 322.126527] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 323.354253] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.967448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 340.982642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 341.008595] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 341.985273] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 341.987370] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 342.013550] Lustre: Skipped 1 previous similar message [ 342.042081] Lustre: Skipped 3 previous similar messages [ 344.957422] Lustre: server umount lustre-MDT0000 complete [ 349.281616] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938078 with bad export cookie 12382190970242767150 [ 349.288427] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.299342] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 352.226264] LustreError: 6536:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.228791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 352.230535] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 352.251367] LustreError: 6536:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 352.277151] Lustre: Skipped 2 previous similar messages [ 354.272276] LustreError: 11882:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.273759] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 354.310061] LustreError: 11882:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 354.332656] Lustre: Skipped 2 previous similar messages [ 356.076233] Lustre: server umount lustre-MDT0001 complete [ 360.385730] Lustre: server umount lustre-OST0000 complete [ 364.667421] Lustre: server umount lustre-OST0001 complete [ 372.099566] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 382.506515] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 423.662262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.017966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.112622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 424.204449] Lustre: lustre-MDT0000: new disk, initializing [ 424.297979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.320797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.340306] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 438.058774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.231175] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.293822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.304409] Lustre: Skipped 1 previous similar message [ 438.397542] Lustre: lustre-MDT0001: new disk, initializing [ 438.497989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.544619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.560790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.778488] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 454.238249] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.601038] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.027536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.287653] Lustre: lustre-OST0000: new disk, initializing [ 471.291230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.302535] Lustre: 17063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.395576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.509417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.549576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.819117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.286907] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 491.078430] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 496.797613] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 497.141095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.657914] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.873449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.415203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 509.416490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 509.426079] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 509.433144] Lustre: Skipped 1 previous similar message [ 509.454057] Lustre: Skipped 1 previous similar message [ 512.347866] Lustre: server umount lustre-OST0000 complete [ 519.654593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 519.666074] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 519.678722] Lustre: Skipped 1 previous similar message [ 530.399279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 530.628273] Lustre: server umount lustre-MDT0000 complete [ 533.988731] LustreError: 15102:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.038563] LustreError: 15102:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 534.365782] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938263 with bad export cookie 12382190970242768690 [ 534.370189] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.382831] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 534.770752] Lustre: server umount lustre-MDT0001 complete [ 539.086391] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:11:06 (1784938266) [ 541.528525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 543.233563] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:11:10 (1784938270) [ 549.656301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.110806] LustreError: 19472:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.219894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.369845] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 555.500462] LustreError: 19473:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.470062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.356871] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 574.728407] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 580.138394] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.600934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.760484] LustreError: 21290:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.798354] LustreError: 21290:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 594.558529] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 601.954325] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 608.737472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 608.745856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 608.769202] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 608.778651] Lustre: Skipped 5 previous similar messages [ 610.799470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 614.860449] Lustre: server umount lustre-OST0000 complete [ 619.491137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 633.312746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 633.518231] Lustre: server umount lustre-MDT0000 complete [ 636.392782] LustreError: 19467:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.412070] LustreError: 19467:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 636.727371] LustreError: 21313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938365 with bad export cookie 12382190970242769572 [ 636.734301] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.736883] LustreError: 21313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 637.092989] Lustre: server umount lustre-MDT0001 complete [ 646.694617] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 649.439298] Key type lgssc unregistered [ 649.742139] LNet: 23006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.749434] LNetError: 23006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.765915] LNet: Removed LNI 192.168.203.114@tcp [ 650.789424] Key type .llcrypt unregistered [ 650.793280] Key type ._llcrypt unregistered [ 671.262852] Key type ._llcrypt registered [ 671.267980] Key type .llcrypt registered [ 671.389718] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 672.272684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.319286] alg: No test for adler32 (adler32-zlib) [ 673.364176] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 673.573697] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 675.288086] Key type lgssc registered [ 676.315540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.666432] loop: module loaded [ 683.779780] loop0: detected capacity change from 0 to 204800000 [ 684.107734] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.124811] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.379976] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.297881] loop0: detected capacity change from 0 to 204800000 [ 689.450774] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.463494] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.475316] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.231646] loop0: detected capacity change from 0 to 204800000 [ 699.735823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 699.766300] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.031377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 701.107743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.222580] Lustre: lustre-MDT0000: new disk, initializing [ 701.377577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.401843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 706.142158] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 716.263232] loop1: detected capacity change from 0 to 204800000 [ 722.080701] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 722.419271] Lustre: lustre-OST07c6: new disk, initializing [ 722.427786] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 722.436632] Lustre: Skipped 1 previous similar message [ 722.449971] Lustre: 26190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 722.545460] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 728.117577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 728.123635] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 728.166503] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 729.170800] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 734.237446] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 734.243476] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 734.254339] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 734.261247] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 734.267041] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 734.274110] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 751.082247] Lustre: 26440:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 751.090424] Lustre: 26440:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.095760] Lustre: 26440:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.105937] Lustre: 26440:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 751.112215] Lustre: 26440:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 751.116511] Lustre: 26440:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.528218] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 755.541326] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.551445] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 755.559192] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.567354] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.574525] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.583954] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 755.590650] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.596075] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 755.601753] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.606681] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.612058] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.781423] Lustre: 26440:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 758.790823] Lustre: 26440:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.795755] Lustre: 26440:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 758.801620] Lustre: 26440:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.807161] Lustre: 26440:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.811814] Lustre: 26440:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.817251] Lustre: 26440:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 758.825089] Lustre: 26440:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.836125] Lustre: 26440:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.843149] Lustre: 26440:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.848944] Lustre: 26440:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.854929] Lustre: 26440:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.241562] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 763.249024] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.253866] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.260141] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.268191] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.275234] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.280191] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 763.285073] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.294376] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.302572] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.315355] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.330764] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.957040] Lustre: 25177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 773.964054] Lustre: 25177:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 773.969294] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 773.975574] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 773.988643] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.995103] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.000531] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 774.006729] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.013241] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 774.023020] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.025877] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.030632] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.318929] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 825.328777] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 825.341740] Lustre: lustre-OST07c6: Not available for connect from 192.168.203.114@tcp (stopping) [ 826.813721] Lustre: server umount lustre-OST07c6 complete [ 836.377541] Lustre: server umount lustre-MDT0000 complete [ 852.247177] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 854.842857] Key type lgssc unregistered [ 855.062915] LNet: 28754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.071432] LNetError: 28754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.102363] LNet: Removed LNI 192.168.203.114@tcp [ 856.973738] Key type .llcrypt unregistered [ 856.979256] Key type ._llcrypt unregistered [ 872.757395] Key type ._llcrypt registered [ 872.759525] Key type .llcrypt registered [ 872.838925] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 873.845940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.887166] alg: No test for adler32 (adler32-zlib) [ 875.041848] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 875.323458] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 877.063598] Key type lgssc registered [ 878.219874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 886.432573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 886.454231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.905826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.949239] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 898.540664] LustreError: 30004:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.560657] LustreError: 30004:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 899.494846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.871953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.115338] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 909.970898] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 914.647714] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.028763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.304975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.854882] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 932.932349] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.629108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 950.243935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 950.255654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 950.284944] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 954.354672] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 954.356785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 954.376783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 956.355860] Lustre: server umount lustre-OST0000 complete [ 961.512718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 961.524237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 961.548318] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 961.559942] Lustre: Skipped 1 previous similar message [ 964.577364] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 964.578208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 969.697270] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 969.702078] Lustre: Skipped 2 previous similar messages [ 974.303157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 974.453836] Lustre: server umount lustre-MDT0000 complete [ 974.820717] LustreError: 32719:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.850127] LustreError: 32719:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 977.652237] LustreError: 32990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938706 with bad export cookie 14298495027423037648 [ 977.656406] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.678725] LustreError: 32990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 978.000390] Lustre: server umount lustre-MDT0001 complete [ 989.200714] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 992.646549] Key type lgssc unregistered [ 992.976554] LNet: 33745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.984565] LNetError: 33745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.009713] LNet: Removed LNI 192.168.203.114@tcp [ 993.827208] Key type .llcrypt unregistered [ 993.834954] Key type ._llcrypt unregistered [ 1015.268801] Key type ._llcrypt registered [ 1015.270691] Key type .llcrypt registered [ 1015.369959] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1016.911821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.952962] alg: No test for adler32 (adler32-zlib) [ 1018.119764] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1018.470983] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1020.183520] Key type lgssc registered [ 1021.223733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.288880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.321294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.862319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.572840] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1040.358548] LustreError: 35014:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.379786] LustreError: 35014:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1044.820357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.154460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.525677] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1056.851389] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.866694] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.650834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1070.045038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.196413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1076.062254] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1084.158087] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1095.653107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1095.660700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1095.670073] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1095.692971] Lustre: Skipped 1 previous similar message [ 1095.704138] Lustre: Skipped 1 previous similar message [ 1099.894584] Lustre: server umount lustre-OST0000 complete [ 1105.890662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1105.919732] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1105.932197] Lustre: Skipped 1 previous similar message [ 1111.008785] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1111.022689] Lustre: Skipped 1 previous similar message [ 1115.108517] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1115.116775] Lustre: Skipped 1 previous similar message [ 1118.175187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1118.398663] Lustre: server umount lustre-MDT0000 complete [ 1120.235494] LustreError: 35015:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.256831] LustreError: 35015:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1122.181736] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938851 with bad export cookie 2444146959153118259 [ 1122.196890] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.828770] Lustre: server umount lustre-MDT0001 complete [ 1133.699908] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1136.513592] Key type lgssc unregistered [ 1136.903560] LNet: 38548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.939709] LNetError: 38548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.978323] LNet: Removed LNI 192.168.203.114@tcp [ 1137.954677] Key type .llcrypt unregistered [ 1137.957669] Key type ._llcrypt unregistered [ 1165.445154] Key type ._llcrypt registered [ 1165.448930] Key type .llcrypt registered [ 1165.559801] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1166.549969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1166.590398] alg: No test for adler32 (adler32-zlib) [ 1167.674697] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1167.930400] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1169.631152] Key type lgssc registered [ 1170.738252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.172559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.204208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.655207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.924523] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1189.857217] LustreError: 39966:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.891365] LustreError: 39966:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1191.751116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.074759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.740466] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1206.056576] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.341696] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.028592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.384355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.531938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1227.044358] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1235.533899] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1243.121373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1243.128607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1243.135386] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1243.148866] Lustre: Skipped 2 previous similar messages [ 1243.168441] Lustre: Skipped 2 previous similar messages [ 1243.864681] Lustre: server umount lustre-MDT0000 complete [ 1247.379301] LustreError: 39946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938976 with bad export cookie 14676400731354710954 [ 1247.384679] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.395708] LustreError: 39946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.228952] LustreError: 39961:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.237298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1248.285077] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 1248.296805] Lustre: Skipped 1 previous similar message [ 1252.261446] LustreError: 39960:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.264308] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 1252.280094] LustreError: 39960:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1252.300923] Lustre: Skipped 1 previous similar message [ 1253.730573] Lustre: server umount lustre-MDT0001 complete [ 1273.893513] Lustre: server umount lustre-OST0000 complete [ 1289.331606] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1292.247509] Key type lgssc unregistered [ 1292.641567] LNet: 43826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.649126] LNetError: 43826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.687124] LNet: Removed LNI 192.168.203.114@tcp [ 1294.628610] Key type .llcrypt unregistered [ 1294.630959] Key type ._llcrypt unregistered [ 1317.551437] Key type ._llcrypt registered [ 1317.553884] Key type .llcrypt registered [ 1317.669642] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1318.778067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.800912] alg: No test for adler32 (adler32-zlib) [ 1319.862551] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1320.118973] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1321.834563] Key type lgssc registered [ 1323.021899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.046649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1330.070938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.535901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.157124] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1340.909189] LustreError: 45245:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.963464] LustreError: 45245:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1344.994666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.429368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.917421] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1358.598979] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.028345] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1372.571388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.031264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.111453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1379.126649] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1387.149185] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.188124] Lustre: lustre-OST0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 1400.804830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1400.813683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1400.822135] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1400.833166] Lustre: Skipped 1 previous similar message [ 1403.091659] Lustre: server umount lustre-OST0000 complete [ 1420.715630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1420.740307] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1420.751280] Lustre: Skipped 2 previous similar messages [ 1422.769682] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1422.775134] Lustre: Skipped 3 previous similar messages [ 1427.128106] Lustre: server umount lustre-MDT0000 complete [ 1427.941810] LustreError: 45244:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.972488] LustreError: 45244:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1431.154483] LustreError: 45225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939160 with bad export cookie 8901999242991088247 [ 1431.178794] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.583678] Lustre: server umount lustre-MDT0001 complete [ 1447.953959] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1451.304770] Key type lgssc unregistered [ 1451.667546] LNet: 48954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.697690] LNetError: 48954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.736292] LNet: Removed LNI 192.168.203.114@tcp [ 1452.572887] Key type .llcrypt unregistered [ 1452.576270] Key type ._llcrypt unregistered [ 1475.587137] Key type ._llcrypt registered [ 1475.590906] Key type .llcrypt registered [ 1475.732024] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1476.704275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.890032] alg: No test for adler32 (adler32-zlib) [ 1477.954270] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1478.198917] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1479.919169] Key type lgssc registered [ 1480.955831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1488.214315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1488.232675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.683021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.186605] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1499.112635] LustreError: 50375:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.126163] LustreError: 50375:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1501.908151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.359112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1506.425987] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1513.064131] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1518.548373] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1524.641759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.981456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.577497] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1534.234716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1537.702551] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1544.584231] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1548.372458] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1561.088434] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-2420h28m07s [ 1562.963949] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1565.174195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1565.189271] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1565.197062] Lustre: Skipped 1 previous similar message [ 1566.057542] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 1566.068299] Lustre: Skipped 2 previous similar messages [ 1567.109428] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 1567.121264] Lustre: Skipped 3 previous similar messages [ 1571.569977] Lustre: server umount lustre-MDT0000 complete [ 1572.164799] LustreError: 50371:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.774657] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1578.315207] LustreError: 50370:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.333654] LustreError: 50370:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 1582.318269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.466871] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.728392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.801493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1583.822211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 1586.918233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1590.791897] Lustre: DEBUG MARKER: Wait for df (36716) ... [ 1592.457903] Lustre: DEBUG MARKER: done [ 1598.433162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1598.444576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1598.480150] Lustre: Skipped 1 previous similar message [ 1598.493940] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1598.959926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1603.912513] Lustre: server umount lustre-OST0000 complete [ 1609.745831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1609.746095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1609.752987] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1609.752996] Lustre: Skipped 3 previous similar messages [ 1609.768323] Lustre: Skipped 2 previous similar messages [ 1614.010925] Lustre: server umount lustre-MDT0000 complete [ 1617.391333] LustreError: 50375:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.425108] LustreError: 50375:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 1618.774555] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939347 with bad export cookie 11379696025809485898 [ 1618.787727] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.797464] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1619.257204] Lustre: server umount lustre-MDT0001 complete [ 1632.028141] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1634.406223] Key type lgssc unregistered [ 1634.757105] LNet: 55297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.762888] LNetError: 55297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.783640] LNet: Removed LNI 192.168.203.114@tcp [ 1635.568440] Key type .llcrypt unregistered [ 1635.570682] Key type ._llcrypt unregistered [ 1671.055232] Key type ._llcrypt registered [ 1671.060527] Key type .llcrypt registered [ 1671.151770] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1672.163692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1672.178815] alg: No test for adler32 (adler32-zlib) [ 1673.241991] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1673.498201] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1675.159219] Key type lgssc registered [ 1676.100233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.705785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1683.729380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.072783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1685.088518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1685.244818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.518316] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1696.665715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.756790] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1696.775412] Lustre: 57506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1696.957087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.851768] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1707.172097] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.034373] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1719.300280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.443390] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1719.622590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.208907] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1726.614745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1732.442113] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1739.861359] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-2420h31m07s [ 1741.340791] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1748.123177] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1750.451481] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1750.523432] LustreError: 59675:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1770.887753] Lustre: lustre-MDT0000: Client 3844aca9-960a-4c78-bb6f-68092fc95bb4 (at 192.168.203.14@tcp) reconnecting [ 1790.567230] LustreError: 59675:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1792.942361] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1795.575078] Lustre: DEBUG MARKER: done [ 1801.936367] Lustre: server umount lustre-OST0000 complete [ 1808.866220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1808.883830] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1808.903372] Lustre: Skipped 1 previous similar message [ 1808.933211] Lustre: Skipped 1 previous similar message [ 1810.022960] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1810.022960] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1816.550526] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1816.717157] Lustre: server umount lustre-MDT0000 complete [ 1819.040144] Lustre: 56832:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939532/real 1784939532] req@0000000088dd8a4c x1871644618446592/t0(0) o13->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:7/4 lens 224/368 e 0 to 1 dl 1784939548 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1819.092518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1820.167859] LustreError: 57498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939549 with bad export cookie 6033822085111545843 [ 1820.174504] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.187934] LustreError: 57498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1820.626991] Lustre: server umount lustre-MDT0001 complete [ 1830.893594] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1833.342702] Key type lgssc unregistered [ 1833.583284] LNet: 61830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.600827] LNetError: 61830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.628306] LNet: Removed LNI 192.168.203.114@tcp [ 1834.446996] Key type .llcrypt unregistered [ 1834.452300] Key type ._llcrypt unregistered [ 1877.442677] Key type ._llcrypt registered [ 1877.446577] Key type .llcrypt registered [ 1877.647755] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1878.747362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.792171] alg: No test for adler32 (adler32-zlib) [ 1879.815295] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1880.094811] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1881.751166] Key type lgssc registered [ 1883.020085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.018377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.071877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.457779] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1887.480670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.640496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.553957] Lustre: Failing over lustre-MDT0000 [ 1888.797346] Lustre: server umount lustre-MDT0000 complete [ 1895.292535] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:33:42 (1784939622) [ 1902.019785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.710530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.809317] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1916.571745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.694508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1916.746178] Lustre: 64628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1917.039862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1921.235855] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1927.288878] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1933.661420] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1940.472907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.813947] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1941.062425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.577918] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1947.931017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1955.640896] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1961.348990] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1968.097103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 1968.109078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1968.125604] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1968.614296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1968.633536] Lustre: Skipped 1 previous similar message [ 1968.645272] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 1968.654093] Lustre: Skipped 1 previous similar message [ 1973.582141] Lustre: server umount lustre-MDT0000 complete [ 1977.328456] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939706 with bad export cookie 3260102863097438493 [ 1977.335125] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.341362] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1977.722398] Lustre: server umount lustre-MDT0001 complete [ 1980.844826] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1985.341713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1994.207171] Lustre: 63550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939707/real 1784939707] req@00000000f47f46d0 x1871644835487616/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1994.240779] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1994.300369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.007491] Lustre: 63551:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939712/real 1784939712] req@00000000608c4734 x1871644835487744/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.353414] Lustre: 63552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939713/real 1784939713] req@000000006386b529 x1871644835487872/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.463877] LustreError: 63549:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000df2d6b96 x1871644835488640/t0(0) o250->MGC192.168.203.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.866886] LustreError: 66475:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.893880] LustreError: 66475:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 2002.962956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.084597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2003.935716] Lustre: 63552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939716/real 1784939716] req@00000000038e9197 x1871644835488128/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2004.015597] LustreError: 66476:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.051653] LustreError: 66476:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2007.187378] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2008.031093] Lustre: 63550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939721/real 1784939721] req@00000000631ec399 x1871644835488256/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2009.064562] LustreError: 68398:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.103114] LustreError: 68398:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2014.187728] LustreError: 66475:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.210533] LustreError: 66475:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2015.929547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.394448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.059518] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2021.869355] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 2022.977505] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 2029.044210] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.172629] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2046.440284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2046.454047] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2046.465311] Lustre: Skipped 1 previous similar message [ 2046.478853] Lustre: Skipped 2 previous similar messages [ 2048.662661] Lustre: server umount lustre-MDT0000 complete [ 2051.558562] LustreError: 68399:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.584953] LustreError: 68399:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2052.924790] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939782 with bad export cookie 3260102863097440768 [ 2052.926586] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2052.937548] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2053.225671] Lustre: server umount lustre-MDT0001 complete [ 2060.108452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2068.382482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.031644] Lustre: 63552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939785/real 1784939785] req@000000008156e45d x1871644835517056/t0(0) o400->lustre-MDT0001-lwp-OST0000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1784939801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.086601] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2072.106594] Lustre: Skipped 1 previous similar message [ 2078.303731] LustreError: 63549:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005691cbd3 x1871644835518080/t0(0) o250->MGC192.168.203.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2078.697163] LustreError: 69132:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.810984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.887904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2083.409966] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2091.507527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.764851] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2097.134296] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 2101.307615] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 2104.023583] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2109.584358] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.591478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2116.596212] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2116.601610] Lustre: Skipped 1 previous similar message [ 2121.702270] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2121.715529] Lustre: Skipped 4 previous similar messages [ 2122.704239] Lustre: server umount lustre-MDT0000 complete [ 2126.426153] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939855 with bad export cookie 3260102863097442147 [ 2126.432481] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.437736] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2126.853281] Lustre: server umount lustre-MDT0001 complete [ 2132.389533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2135.356774] Lustre: DEBUG MARKER: files compared the same [ 2147.516380] Lustre: server umount lustre-OST0000 complete [ 2162.802068] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2165.516459] Key type lgssc unregistered [ 2165.795965] LNet: 74355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.802151] LNetError: 74355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.816971] LNet: Removed LNI 192.168.203.114@tcp [ 2166.608419] Key type .llcrypt unregistered [ 2166.614132] Key type ._llcrypt unregistered [ 2189.868789] Key type ._llcrypt registered [ 2189.871073] Key type .llcrypt registered [ 2189.982440] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2191.170849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.199889] alg: No test for adler32 (adler32-zlib) [ 2192.263900] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2192.516469] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2194.232412] Key type lgssc registered [ 2195.458471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.024908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.076593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.469172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.676597] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug +malloc all [ 2214.887323] LustreError: 75772:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.928024] LustreError: 75772:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2218.685882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.074129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.524790] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug +malloc all [ 2231.474917] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2237.200978] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2243.735645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.103331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.172510] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug +malloc all [ 2250.298623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2257.505345] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.841742] Lustre: server umount lustre-OST0000 complete [ 2270.177705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2270.186220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2270.208086] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2270.688748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2275.808528] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2275.817337] Lustre: Skipped 2 previous similar messages [ 2278.933259] Lustre: server umount lustre-MDT0000 complete [ 2280.927436] Lustre: 75085:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939994/real 1784939994] req@00000000bc5a2f00 x1871645162629376/t0(0) o13->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:7/4 lens 224/368 e 0 to 1 dl 1784940010 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2280.930318] LustreError: 75769:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.965534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2280.999554] LustreError: 75769:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2281.503309] Lustre: 75087:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939994/real 1784939994] req@000000008fb764ab x1871645162629120/t0(0) o400->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:28/4 lens 224/224 e 0 to 1 dl 1784940010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2282.008087] LustreError: 75754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940011 with bad export cookie 9696685538428181952 [ 2282.008566] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.023403] LustreError: 75754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2282.313189] Lustre: server umount lustre-MDT0001 complete [ 2291.186908] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2294.047873] Key type lgssc unregistered [ 2294.308517] LNet: 79304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.321043] LNetError: 79304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.335353] LNet: Removed LNI 192.168.203.114@tcp [ 2295.178433] Key type .llcrypt unregistered [ 2295.182974] Key type ._llcrypt unregistered [ 2318.075657] Key type ._llcrypt registered [ 2318.079851] Key type .llcrypt registered [ 2318.167982] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2319.505253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.537924] alg: No test for adler32 (adler32-zlib) [ 2320.541515] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2320.749978] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2322.432758] Key type lgssc registered [ 2323.817371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.481935] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.504431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2347.039770] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2352.159379] LustreError: 80695:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 2378.783175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.804957] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2393.947486] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.206478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.341065] Lustre: *** cfs_fail_loc=706, val=0*** [ 2402.742392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.822392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2406.855775] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2407.908845] LustreError: 80721:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.930979] LustreError: 80721:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2413.032447] LustreError: 80722:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.059153] LustreError: 80722:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2414.695260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.933996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2418.601466] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2426.043209] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2431.958227] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2435.567211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2435.571848] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2435.583780] Lustre: Skipped 1 previous similar message [ 2435.597064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2440.694951] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2440.701924] Lustre: Skipped 1 previous similar message [ 2445.806134] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2445.824395] Lustre: Skipped 2 previous similar messages [ 2448.351245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2448.503248] Lustre: server umount lustre-OST0000 complete [ 2456.036409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2456.047797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2456.068535] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2456.096056] Lustre: Skipped 1 previous similar message [ 2458.569102] Lustre: server umount lustre-MDT0000 complete [ 2461.157878] LustreError: 83034:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.183669] LustreError: 83034:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2461.944118] LustreError: 80703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940191 with bad export cookie 12470625445976049712 [ 2461.953378] LustreError: 80703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2461.957330] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2462.346644] Lustre: server umount lustre-MDT0001 complete [ 2472.781446] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2475.641139] Key type lgssc unregistered [ 2475.938217] LNet: 84310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.955265] LNetError: 84310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.971728] LNet: Removed LNI 192.168.203.114@tcp [ 2476.903451] Key type .llcrypt unregistered [ 2476.905824] Key type ._llcrypt unregistered [ 2498.453376] Key type ._llcrypt registered [ 2498.456161] Key type .llcrypt registered [ 2498.579995] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2499.637031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.806802] alg: No test for adler32 (adler32-zlib) [ 2500.884704] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2501.141209] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2502.815160] Key type lgssc registered [ 2503.976588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.029723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.044328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.137429] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2522.539382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.919790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2527.194487] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2528.228381] LustreError: 86392:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.243450] LustreError: 86392:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 2533.350166] LustreError: 86391:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.366531] LustreError: 86391:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2534.397736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.686296] LustreError: 86395:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.710974] LustreError: 86395:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2534.734706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.008431] LustreError: 86396:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.037826] LustreError: 86396:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2540.659151] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2548.137666] LustreError: 86392:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.158267] LustreError: 86392:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 2548.647467] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.665927] Lustre: MGS: Client 65e725cc-6522-4567-9217-920e931b6b38 (at 0@lo) reconnecting [ 2548.691461] LustreError: 86406:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2548.915826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.001496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2552.586923] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2568.684931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2568.690892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2568.700846] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2569.702165] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2569.710633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2569.725584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2571.779262] Lustre: server umount lustre-OST0000 complete [ 2579.937985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2579.941197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2579.955381] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2579.971815] Lustre: Skipped 2 previous similar messages [ 2581.232040] Lustre: server umount lustre-MDT0000 complete [ 2585.094533] Lustre: server umount lustre-MDT0001 complete [ 2588.906719] Lustre: server umount MGS complete [ 2600.460802] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:45:27 (1784940327) [ 2629.519931] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 2638.743016] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2669.750383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.127110] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2683.569730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.854409] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2683.912748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2683.983671] Lustre: lustre-MDT0001: new disk, initializing [ 2684.077918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.112535] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2684.122937] Lustre: 93559:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2684.137487] LustreError: 93559:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2688.009583] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2699.673494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.991535] Lustre: lustre-OST0000: new disk, initializing [ 2699.994438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2700.011673] Lustre: 94326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2700.116946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.180604] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2706.596581] Lustre: 94624:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000006f4b7c23 x1871645486651904/t0(0) o700->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:31/4 lens 264/248 e 0 to 0 dl 1784940451 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2715.456903] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.243205] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.258034] LustreError: 85063:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8d4e531 x1871645486653312/t0(0) o250->MGC192.168.203.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2721.279492] Lustre: MGS: Client 072756fc-b37f-42a1-9e05-ed3eda21a2f0 (at 0@lo) reconnecting [ 2721.356171] Lustre: 92828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2721.377384] LustreError: 92828:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2721.391402] LustreError: 92828:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2721.410311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2721.481683] Lustre: lustre-MDT0000: new disk, initializing [ 2721.613592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.630879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2721.648624] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2722.725654] Lustre: 95396:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000cf6465e9 x1871645486658688/t0(0) o700->lustre-MDT0001-osp-MDT0000@192.168.203.114@tcp:30/10 lens 264/248 e 0 to 0 dl 1784940467 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2725.473449] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2727.840732] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2731.818668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2731.837184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2731.918416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2757.621066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2757.629275] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2757.644261] Lustre: Skipped 2 previous similar messages [ 2757.649286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2761.635596] Lustre: server umount lustre-OST0000 complete [ 2767.843085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2767.886057] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2767.902052] Lustre: Skipped 2 previous similar messages [ 2771.691839] Lustre: server umount lustre-MDT0000 complete [ 2772.971183] LustreError: 93534:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.000129] LustreError: 93534:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2781.918198] Lustre: server umount lustre-MDT0001 complete [ 2785.790656] Lustre: server umount MGS complete [ 2797.988672] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:48:44 (1784940524) [ 2817.719476] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2820.310942] Key type lgssc unregistered [ 2820.664814] LNet: 97892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.679793] LNetError: 97892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2820.738141] LNet: Removed LNI 192.168.203.114@tcp [ 2821.632699] Key type .llcrypt unregistered [ 2821.639787] Key type ._llcrypt unregistered [ 2839.823411] Key type ._llcrypt registered [ 2839.825589] Key type .llcrypt registered [ 2839.977334] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2841.443834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2841.511144] alg: No test for adler32 (adler32-zlib) [ 2842.543994] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2842.807899] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2844.479343] Key type lgssc registered [ 2845.508230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2852.533292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2852.572335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.157773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.565171] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2864.612916] LustreError: 99112:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.636021] LustreError: 99112:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2866.757731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.151075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.053354] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2879.295830] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2885.351914] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2892.155276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.541738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.484446] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2899.728260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2906.682959] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2915.318683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2915.324369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2915.335509] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2915.350328] Lustre: Skipped 1 previous similar message [ 2917.009607] Lustre: server umount lustre-OST0000 complete [ 2923.497897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2923.504372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2923.516273] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2923.520871] Lustre: Skipped 1 previous similar message [ 2925.538955] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 2925.541549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2926.951662] Lustre: server umount lustre-MDT0000 complete [ 2930.284699] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940659 with bad export cookie 11360122383724169077 [ 2930.290447] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2930.293136] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2930.727700] Lustre: server umount lustre-MDT0001 complete [ 2940.686550] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2943.309090] Key type lgssc unregistered [ 2943.661323] LNet: 102647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2943.670214] LNetError: 102647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2943.687187] LNet: Removed LNI 192.168.203.114@tcp [ 2944.791398] Key type .llcrypt unregistered [ 2944.793743] Key type ._llcrypt unregistered [ 2961.903751] Key type ._llcrypt registered [ 2961.910350] Key type .llcrypt registered [ 2962.072902] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2963.115825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2963.230909] alg: No test for adler32 (adler32-zlib) [ 2964.352732] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2964.704343] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2966.367748] Key type lgssc registered [ 2967.590862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.122522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.125867] LustreError: 103730:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2973.130172] Lustre: Skipped 1 previous similar message [ 2973.130192] LustreError: 103731:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2973.151946] LustreError: 103730:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2973.153091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.162248] LustreError: 103730:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2973.162274] LustreError: 103730:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2974.725810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.950151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.212442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2986.666754] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2991.773574] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing lsmod [ 2994.420166] LustreError: 104938:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2994.440834] LustreError: 104939:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2994.444925] LustreError: 104938:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2994.452882] LustreError: 104938:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2994.462274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2994.474714] LustreError: 104938:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2994.862168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.729335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 2997.742288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2997.766355] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3000.294798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3000.305644] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3000.313560] Lustre: Skipped 1 previous similar message [ 3000.320935] Lustre: Skipped 1 previous similar message [ 3003.732457] Lustre: server umount lustre-MDT0000 complete [ 3005.409659] LustreError: 103763:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.440798] LustreError: 103763:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 3007.133398] LustreError: 103744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940736 with bad export cookie 11948423185247225511 [ 3007.134991] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.138582] LustreError: 103744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3007.157691] Lustre: 103760:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3007.490184] Lustre: server umount lustre-MDT0001 complete [ 3020.247997] Lustre: server umount lustre-OST0000 complete [ 3027.552734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.082037] LustreError: 106002:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.146270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.092633] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3033.578314] LustreError: 106003:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.703535] LustreError: 106002:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.086085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.446196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3044.929695] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3052.537289] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3058.039962] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3064.666493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.985717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.445652] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3073.221729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3079.345255] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3086.821672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3086.839864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3086.855660] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3086.862244] Lustre: Skipped 1 previous similar message [ 3088.004153] Lustre: server umount lustre-OST0000 complete [ 3091.936886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3091.947624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3091.966792] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3099.116106] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3099.127223] Lustre: Skipped 3 previous similar messages [ 3101.216474] Lustre: server umount lustre-MDT0000 complete [ 3104.234120] LustreError: 107024:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.265065] LustreError: 107024:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3104.551903] LustreError: 105982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940833 with bad export cookie 11948423185247226183 [ 3104.552620] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.568057] LustreError: 105982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3104.896342] Lustre: server umount lustre-MDT0001 complete [ 3114.817300] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3117.472229] Key type lgssc unregistered [ 3117.760937] LNet: 109531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3117.771330] LNetError: 109531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3117.790852] LNet: Removed LNI 192.168.203.114@tcp [ 3118.556808] Key type .llcrypt unregistered [ 3118.558628] Key type ._llcrypt unregistered [ 3140.270917] Key type ._llcrypt registered [ 3140.273281] Key type .llcrypt registered [ 3140.342399] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3140.989895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3141.016074] alg: No test for adler32 (adler32-zlib) [ 3142.124158] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3142.333958] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3143.984729] Key type lgssc registered [ 3145.055815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3152.137028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3152.154827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.605783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.572260] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3163.109067] LustreError: 110947:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.122949] LustreError: 110947:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3166.069899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.495272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3171.240512] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3178.154958] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.188758] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3189.874633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.259540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.227395] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3198.470232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3203.684566] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3208.409127] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3211.698143] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3216.874105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3216.885395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3216.904843] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3217.985814] Lustre: server umount lustre-OST0000 complete [ 3223.014369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3223.033536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3223.066440] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3229.152146] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3229.166422] Lustre: Skipped 3 previous similar messages [ 3234.274284] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3234.278457] Lustre: Skipped 1 previous similar message [ 3236.319140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3236.592272] Lustre: server umount lustre-MDT0000 complete [ 3239.395845] LustreError: 110948:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.430491] LustreError: 110948:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3240.569219] LustreError: 110928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940969 with bad export cookie 12769041343666825963 [ 3240.572378] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.600259] LustreError: 110928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3241.002603] Lustre: server umount lustre-MDT0001 complete [ 3253.356216] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3257.170754] Key type lgssc unregistered [ 3257.558641] LNet: 114587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3257.576732] LNetError: 114587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3257.603431] LNet: Removed LNI 192.168.203.114@tcp [ 3258.739443] Key type .llcrypt unregistered [ 3258.743237] Key type ._llcrypt unregistered [ 3276.192061] Key type ._llcrypt registered [ 3276.194114] Key type .llcrypt registered [ 3276.330245] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3277.329257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.403994] alg: No test for adler32 (adler32-zlib) [ 3278.517706] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3278.789750] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3280.448290] Key type lgssc registered [ 3281.464558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3288.636796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3288.653509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.205755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.507919] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3300.324667] LustreError: 115809:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3300.347056] LustreError: 115809:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3302.708707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.020748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3307.296231] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3315.647903] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3321.299075] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3327.912142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.379123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.781333] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3335.550161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3342.373156] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3359.225504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3359.229374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3359.237637] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3359.266692] Lustre: Skipped 1 previous similar message [ 3363.105973] Lustre: server umount lustre-OST0000 complete [ 3369.446503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3369.448509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3369.505072] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3369.516541] Lustre: Skipped 2 previous similar messages [ 3374.502968] Lustre: server umount lustre-MDT0000 complete [ 3374.564819] LustreError: 115810:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.583647] LustreError: 115810:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3378.353320] LustreError: 115790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941107 with bad export cookie 7879963953844078285 [ 3378.355885] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.381307] LustreError: 115790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3378.960728] Lustre: server umount lustre-MDT0001 complete [ 3391.093292] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3393.944217] Key type lgssc unregistered [ 3394.380344] LNet: 119361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.393920] LNetError: 119361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.418030] LNet: Removed LNI 192.168.203.114@tcp [ 3395.382947] Key type .llcrypt unregistered [ 3395.385045] Key type ._llcrypt unregistered [ 3420.104979] Key type ._llcrypt registered [ 3420.109747] Key type .llcrypt registered [ 3420.242676] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3421.244542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3421.259656] alg: No test for adler32 (adler32-zlib) [ 3422.321196] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3422.642865] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3424.343756] Key type lgssc registered [ 3425.522286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3433.430550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3433.452866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3434.803294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.977171] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3444.197912] LustreError: 120781:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.223982] LustreError: 120781:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3446.425403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.759471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3450.705610] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3457.637296] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3462.628089] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.105531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.648977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.997911] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3478.960866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3485.389300] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3491.993481] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3495.661896] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3495.671948] Lustre: Skipped 2 previous similar messages [ 3500.104824] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3500.110441] Lustre: Skipped 2 previous similar messages [ 3501.641418] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3501.673723] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3511.311233] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3511.318425] Lustre: Skipped 2 previous similar messages [ 3511.823333] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.14@tcp 192.168.0.[2,10]@tcp [ 3517.806120] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.14/24@tcp in log lustre-MDT0000 [ 3517.814176] Lustre: Skipped 2 previous similar messages [ 3528.289562] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.14/24@tcp [ 3528.301257] Lustre: Skipped 1 previous similar message [ 3535.334461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3535.350478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3535.363662] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3535.378672] Lustre: Skipped 1 previous similar message [ 3540.458240] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3540.475120] Lustre: Skipped 1 previous similar message [ 3545.569976] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3549.151116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3549.380499] Lustre: server umount lustre-OST0000 complete [ 3554.273230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3554.302681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3554.326909] Lustre: Skipped 1 previous similar message [ 3554.334491] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3554.345253] Lustre: Skipped 1 previous similar message [ 3555.821415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3560.024577] Lustre: server umount lustre-MDT0000 complete [ 3560.939561] LustreError: 120780:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.984596] LustreError: 120780:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3564.638936] LustreError: 120762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941293 with bad export cookie 11903277578307419110 [ 3564.642771] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.654686] LustreError: 120762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3565.123956] Lustre: server umount lustre-MDT0001 complete [ 3575.304734] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3578.104659] Key type lgssc unregistered [ 3578.450651] LNet: 125637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3578.460894] LNetError: 125637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3578.492286] LNet: Removed LNI 192.168.203.114@tcp [ 3579.622650] Key type .llcrypt unregistered [ 3579.625257] Key type ._llcrypt unregistered [ 3611.981248] Key type ._llcrypt registered [ 3611.989843] Key type .llcrypt registered [ 3612.104524] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3613.251840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.273905] alg: No test for adler32 (adler32-zlib) [ 3614.417614] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3614.661513] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3616.351501] Key type lgssc registered [ 3617.495515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.426710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.446268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.894949] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3626.935189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.940116] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.14/24@tcp [ 3630.809571] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3637.247719] LustreError: 127202:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.280597] LustreError: 127202:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 3638.563857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.840120] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3638.861852] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.14/24@tcp [ 3638.937227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3643.172409] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3649.310748] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3653.747610] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.206576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3660.542196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.086502] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3672.041628] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3672.828033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3683.301343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3683.314809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3683.343101] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3683.354798] Lustre: Skipped 1 previous similar message [ 3688.236801] Lustre: server umount lustre-OST0000 complete [ 3693.538648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3693.558683] Lustre: Skipped 2 previous similar messages [ 3693.561497] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3693.577835] Lustre: Skipped 1 previous similar message [ 3697.661754] Lustre: server umount lustre-MDT0000 complete [ 3698.660800] LustreError: 127202:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.678815] LustreError: 127202:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3700.807610] LustreError: 127183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941429 with bad export cookie 12788220106143358471 [ 3700.811862] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3700.824629] LustreError: 127183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3701.131443] Lustre: server umount lustre-MDT0001 complete [ 3709.884403] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3711.993866] Key type lgssc unregistered [ 3712.218984] LNet: 130789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.227712] LNetError: 130789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.243718] LNet: Removed LNI 192.168.203.114@tcp [ 3713.037202] Key type .llcrypt unregistered [ 3713.040484] Key type ._llcrypt unregistered [ 3734.843717] Key type ._llcrypt registered [ 3734.845813] Key type .llcrypt registered [ 3735.021238] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3750.632115] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3751.592553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3751.763179] alg: No test for adler32 (adler32-zlib) [ 3752.908501] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 3753.206649] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3754.839689] Key type lgssc registered [ 3756.050354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3785.731178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3785.767880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.995392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3787.026769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3787.098547] Lustre: lustre-MDT0000: new disk, initializing [ 3787.164953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3787.187562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3791.120420] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3800.568538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.713554] Lustre: 134190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3800.775794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3800.781269] Lustre: Skipped 1 previous similar message [ 3800.879021] Lustre: lustre-MDT0001: new disk, initializing [ 3801.006086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.069383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3801.084073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3804.964596] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3812.571795] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3817.834157] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.235759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.435755] Lustre: lustre-OST0000: new disk, initializing [ 3823.438866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3823.449935] Lustre: 136160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3823.540984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3828.718451] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3829.841553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3829.852537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3829.989921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3836.386311] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3840.104615] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3840.272375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3843.771683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3843.976320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3845.606897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3845.630954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3845.656986] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3850.723310] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3850.734427] Lustre: Skipped 2 previous similar messages [ 3851.652339] Lustre: server umount lustre-OST0000 complete [ 3855.853235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3855.856359] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3855.872250] Lustre: Skipped 2 previous similar messages [ 3855.882793] Lustre: Skipped 1 previous similar message [ 3860.730725] Lustre: server umount lustre-MDT0000 complete [ 3863.551230] LustreError: 134183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941592 with bad export cookie 2652779124159382568 [ 3863.554021] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.557822] LustreError: 134183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3863.884888] Lustre: server umount lustre-MDT0001 complete [ 3871.315476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.847630] LustreError: 138375:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.872102] LustreError: 138375:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 3871.927268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.405438] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3877.351357] LustreError: 138376:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.475510] LustreError: 138375:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.367385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.741721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3888.609365] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3895.356656] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3899.879675] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3905.380075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.693230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.144926] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3917.104709] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3920.807684] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3921.003964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3924.811606] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3924.989523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3931.489975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.612627] Lustre: lustre-OST0001: new disk, initializing [ 3931.616123] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3931.622740] Lustre: 141745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3931.705894] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.864199] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3941.447676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3941.464546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3941.542710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3943.525453] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3943.712928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3947.307841] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3947.520972] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3951.731769] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92326055d800.ost_server_uuid 50 [ 3952.852396] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92326055d800.ost_server_uuid in FULL state after 0 sec [ 3962.347385] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3962.347851] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3962.358027] LustreError: Skipped 2 previous similar messages [ 3962.381858] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 3962.393365] Lustre: Skipped 1 previous similar message [ 3965.417551] Lustre: server umount lustre-OST0001 complete [ 3967.467247] LustreError: 140196:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.497156] LustreError: 140196:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3968.481929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3968.515109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3968.527313] Lustre: Skipped 1 previous similar message [ 3968.534642] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3971.554384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 3972.582930] LustreError: 140193:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.599651] LustreError: 140193:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 3974.666431] Lustre: server umount lustre-OST0000 complete [ 3977.700091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3977.708647] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 3977.709601] Lustre: Skipped 2 previous similar messages [ 3977.722812] Lustre: Skipped 4 previous similar messages [ 3983.586299] Lustre: server umount lustre-MDT0000 complete [ 3986.479123] LustreError: 138355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941715 with bad export cookie 2652779124159383450 [ 3986.480130] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.501314] LustreError: 138355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3986.770203] Lustre: server umount lustre-MDT0001 complete [ 4000.890485] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4003.184500] Key type lgssc unregistered [ 4003.393488] LNet: 144746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.400111] LNetError: 144746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.419127] LNet: Removed LNI 192.168.203.114@tcp [ 4004.076988] Key type .llcrypt unregistered [ 4004.079868] Key type ._llcrypt unregistered [ 4031.641621] Key type ._llcrypt registered [ 4031.643581] Key type .llcrypt registered [ 4031.746717] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4041.974733] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4042.655215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4042.688244] alg: No test for adler32 (adler32-zlib) [ 4043.691726] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4043.869912] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4045.503181] Key type lgssc registered [ 4046.289946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4068.656637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4068.671328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.812208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4069.834404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4069.922995] Lustre: lustre-MDT0000: new disk, initializing [ 4070.019843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.039612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4072.842860] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4081.207646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.319828] Lustre: 148877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4081.358439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4081.364801] Lustre: Skipped 1 previous similar message [ 4081.435749] Lustre: lustre-MDT0001: new disk, initializing [ 4081.512516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4081.556528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4081.570593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4084.492574] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4091.089287] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4095.070570] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4100.143359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4100.412546] Lustre: lustre-OST0000: new disk, initializing [ 4100.416659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4100.426829] Lustre: 150843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4100.494423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.826257] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4109.348208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4109.355069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4109.442711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4111.531110] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4115.534859] Lustre: Failing over lustre-OST0000 [ 4115.642921] Lustre: server umount lustre-OST0000 complete [ 4131.104719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.369834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4131.382922] Lustre: Skipped 1 previous similar message [ 4131.411023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4131.431685] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4134.765310] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4134.879337] Lustre: 147101:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784941848/real 1784941848] req@00000000d6a8be3b x1871647104595840/t0(0) o6->lustre-OST0000-osc-MDT0000@192.168.203.114@tcp:28/4 lens 544/432 e 0 to 1 dl 1784941864 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4135.251743] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4135.455496] Lustre: 147103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784941848/real 1784941848] req@0000000023762fc8 x1871647104596096/t0(0) o13->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:7/4 lens 224/368 e 0 to 1 dl 1784941864 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4136.468458] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4136.469273] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 4137.960065] Lustre: Failing over lustre-MDT0000 [ 4138.154136] Lustre: server umount lustre-MDT0000 complete [ 4140.000602] Lustre: 147100:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784941853/real 1784941853] req@00000000299fb559 x1871647104597760/t0(0) o400->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:28/4 lens 224/224 e 0 to 1 dl 1784941869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4141.543483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4141.557367] Lustre: Skipped 1 previous similar message [ 4144.966432] LustreError: 151263:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.978262] LustreError: 151263:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 4145.119840] Lustre: 147100:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784941858/real 1784941858] req@000000000280564c x1871647104598912/t0(0) o400->lustre-OST0000-osc-MDT0001@192.168.203.114@tcp:28/4 lens 224/224 e 0 to 1 dl 1784941874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4146.660371] LustreError: 148884:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.679518] LustreError: 148884:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4150.081516] LustreError: 148886:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.821861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.900818] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.908465] Lustre: 148881:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4154.085787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.123975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4155.207479] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4157.150358] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4159.470809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 4159.474781] Lustre: Skipped 1 previous similar message [ 4159.492645] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4159.550384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4164.588875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4164.590526] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4164.600252] Lustre: Skipped 2 previous similar messages [ 4164.610307] Lustre: Skipped 1 previous similar message [ 4165.695717] Lustre: server umount lustre-OST0000 complete [ 4169.700236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4169.703727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4169.714017] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4169.720560] Lustre: Skipped 1 previous similar message [ 4174.230350] LustreError: 153743:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000005a9ac3db) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4174.818734] LustreError: 150039:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.827969] LustreError: 150039:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 4176.315466] Lustre: server umount lustre-MDT0000 complete [ 4178.646581] LustreError: 148914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941907 with bad export cookie 3280742607148797074 [ 4178.647261] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.659503] LustreError: 148914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4178.863571] Lustre: server umount lustre-MDT0001 complete [ 4185.475862] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4187.398457] Key type lgssc unregistered [ 4187.592505] LNet: 154494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4187.601869] LNetError: 154494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4187.617277] LNet: Removed LNI 192.168.203.114@tcp [ 4188.197245] Key type .llcrypt unregistered [ 4188.199666] Key type ._llcrypt unregistered [ 4215.064507] Key type ._llcrypt registered [ 4215.066305] Key type .llcrypt registered [ 4215.140553] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4215.717651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4215.727890] alg: No test for adler32 (adler32-zlib) [ 4216.679629] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4216.849320] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4218.471495] Key type lgssc registered [ 4219.207780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4223.979641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4223.992437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.257452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.668343] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4233.382773] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4235.748225] LustreError: 156013:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.766459] LustreError: 156013:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 4236.251266] LustreError: 156042:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4236.258333] LustreError: 156042:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4236.269270] LustreError: 156042:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4236.480435] Lustre: server umount lustre-MDT0000 complete [ 4245.695243] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4251.511568] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4274.120447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.270886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4274.295601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4274.361947] Lustre: lustre-MDT0000: new disk, initializing [ 4274.421580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.440267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4277.041666] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4284.142542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.233182] Lustre: 159340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4284.286580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4284.290094] Lustre: Skipped 1 previous similar message [ 4284.340793] Lustre: lustre-MDT0001: new disk, initializing [ 4284.398735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.434090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.443993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4287.315909] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4293.727935] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4297.251512] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4301.803952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.023812] Lustre: lustre-OST0000: new disk, initializing [ 4302.028609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4302.034162] Lustre: 161308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.110874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.104274] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4308.547593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4308.558253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4308.636543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4312.594180] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4315.247935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4316.363049] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:14:04 (1784942044) [ 4472.800517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4472.803515] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4472.813934] Lustre: Skipped 2 previous similar messages [ 4472.823901] Lustre: Skipped 2 previous similar messages [ 4475.794302] Lustre: server umount lustre-MDT0000 complete [ 4477.612905] LustreError: 159332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942206 with bad export cookie 4275143954580348778 [ 4477.613882] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.621827] LustreError: 159332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4477.921815] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4477.922370] LustreError: 163071:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.929252] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 4483.040825] LustreError: 159349:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.040833] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 4483.887774] Lustre: server umount lustre-MDT0001 complete [ 4492.094596] Lustre: server umount lustre-OST0000 complete [ 4497.210995] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4502.534081] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4522.649836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.769234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4522.786174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4522.835785] Lustre: lustre-MDT0000: new disk, initializing [ 4522.891822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.903615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4525.208807] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4531.336582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.403702] Lustre: 165948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.443501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4531.446292] Lustre: Skipped 1 previous similar message [ 4531.502312] Lustre: lustre-MDT0001: new disk, initializing [ 4531.573594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4531.598855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4531.610775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4533.942127] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4539.351901] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4542.472715] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4546.260970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4546.448246] Lustre: lustre-OST0000: new disk, initializing [ 4546.455372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4546.460220] Lustre: 167917:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4549.568305] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4554.963911] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4555.280260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4555.287345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4555.353253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4557.330434] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4557.435740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4559.588653] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.703287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4565.479344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4565.479347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4565.482792] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4565.487557] Lustre: Skipped 1 previous similar message [ 4566.830499] Lustre: server umount lustre-OST0000 complete [ 4570.593454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4570.596586] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4570.600800] Lustre: Skipped 1 previous similar message [ 4570.605896] Lustre: Skipped 1 previous similar message [ 4574.804581] Lustre: server umount lustre-MDT0000 complete [ 4575.712479] LustreError: 167088:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.723026] LustreError: 167088:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 4576.799213] LustreError: 165940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942305 with bad export cookie 4275143954580507608 [ 4576.804798] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4576.807529] LustreError: 165940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4576.990746] Lustre: server umount lustre-MDT0001 complete [ 4581.443151] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:18:29 (1784942309) [ 4589.960795] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4595.010967] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4612.698034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.836979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4612.855763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4612.911204] Lustre: lustre-MDT0000: new disk, initializing [ 4612.950759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.954297] Lustre: Skipped 1 previous similar message [ 4612.961139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4615.181818] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4620.895576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.953835] Lustre: 172678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4620.986335] LustreError: 172687:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.097680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4623.111639] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4628.179571] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4631.029671] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4634.576402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.734622] Lustre: 174653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4637.849339] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4639.769868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4639.840470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4643.061300] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4649.955862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4649.956397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4649.961181] LustreError: Skipped 1 previous similar message [ 4649.970619] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4649.975761] Lustre: Skipped 2 previous similar messages [ 4652.669447] Lustre: server umount lustre-OST0000 complete [ 4660.834286] Lustre: server umount lustre-MDT0000 complete [ 4662.824170] LustreError: 172670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942391 with bad export cookie 4275143954580508490 [ 4662.829841] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4662.835507] LustreError: 172670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4663.031080] Lustre: server umount lustre-MDT0001 complete [ 4667.312295] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:19:55 (1784942395) [ 4675.371112] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4680.100089] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4698.093230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.221767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4698.242580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4698.248192] Lustre: Skipped 3 previous similar messages [ 4698.307188] Lustre: lustre-MDT0000: new disk, initializing [ 4698.309873] Lustre: Skipped 2 previous similar messages [ 4698.356506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.360522] Lustre: Skipped 2 previous similar messages [ 4698.374953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4698.381100] Lustre: Skipped 2 previous similar messages [ 4700.656979] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4706.735188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.816409] Lustre: 178965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4706.971178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4708.967225] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4714.231961] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4716.870953] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4720.068487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4720.208075] Lustre: 180941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4722.953078] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4726.882712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4728.133596] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4731.874968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4731.875345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4731.880029] LustreError: Skipped 1 previous similar message [ 4731.885685] Lustre: Skipped 3 previous similar messages [ 4731.887247] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4731.897601] Lustre: Skipped 5 previous similar messages [ 4733.683591] Lustre: server umount lustre-OST0000 complete [ 4750.303261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4750.388941] Lustre: server umount lustre-MDT0000 complete [ 4752.288990] LustreError: 178957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942481 with bad export cookie 4275143954580509575 [ 4752.289472] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.296023] LustreError: 178957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4752.461681] Lustre: server umount lustre-MDT0001 complete [ 4757.449086] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4758.782766] Key type lgssc unregistered [ 4758.935529] LNet: 182797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4758.939867] LNetError: 182797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4758.949197] LNet: Removed LNI 192.168.203.114@tcp [ 4759.530832] Key type .llcrypt unregistered [ 4759.533088] Key type ._llcrypt unregistered [ 4771.595579] Key type ._llcrypt registered [ 4771.597736] Key type .llcrypt registered [ 4771.657053] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4772.063742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4772.118058] alg: No test for adler32 (adler32-zlib) [ 4773.049751] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4773.177363] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4774.791263] Key type lgssc registered [ 4775.430710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4779.646328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4779.663782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.898387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.014437] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4787.511563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.719812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.862076] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4793.864472] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.839392] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4800.311477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.513627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.558960] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4806.700856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4807.242758] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4827.105380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4827.112290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4827.124374] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4828.133177] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4828.135425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4828.151949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4831.085273] Lustre: server umount lustre-OST0000 complete [ 4833.248820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4833.250310] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4833.262029] Lustre: Skipped 1 previous similar message [ 4833.267857] Lustre: Skipped 1 previous similar message [ 4838.373726] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4838.378930] Lustre: Skipped 2 previous similar messages [ 4839.394910] Lustre: server umount lustre-MDT0000 complete [ 4841.502422] LustreError: 184172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942570 with bad export cookie 451454394076564100 [ 4841.505620] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4841.511218] LustreError: 184172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4841.667475] Lustre: server umount lustre-MDT0001 complete [ 4847.364595] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4849.041705] Key type lgssc unregistered [ 4849.239705] LNet: 187739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4849.249455] LNetError: 187739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4849.258404] LNet: Removed LNI 192.168.203.114@tcp [ 4849.763227] Key type .llcrypt unregistered [ 4849.764632] Key type ._llcrypt unregistered [ 4863.610693] Key type ._llcrypt registered [ 4863.613658] Key type .llcrypt registered [ 4863.684752] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4864.189762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4864.202960] alg: No test for adler32 (adler32-zlib) [ 4865.125556] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 4865.271336] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4866.895679] Key type lgssc registered [ 4867.581361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4871.862242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4871.882341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.092705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.192983] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4879.510092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.724047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4882.344790] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4886.184136] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4889.068600] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4892.406982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4892.605685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.426913] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4898.735318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4899.117903] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.026487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4909.031095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4909.041582] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4910.052060] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4910.052414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4910.062043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4911.958141] Lustre: lustre-OST0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 4911.964019] Lustre: Skipped 1 previous similar message [ 4914.090535] Lustre: server umount lustre-OST0000 complete [ 4917.958656] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9232459a7800.ost_server_uuid 50 [ 4918.812956] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4930.531460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4930.534403] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4930.540276] Lustre: Skipped 1 previous similar message [ 4930.558613] Lustre: Skipped 1 previous similar message [ 4936.674856] LustreError: 189133:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.686983] LustreError: 189133:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 4936.720824] Lustre: server umount lustre-MDT0000 complete [ 4938.805771] LustreError: 189114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942667 with bad export cookie 11657684985270450287 [ 4938.810192] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.813422] LustreError: 189114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4939.001667] Lustre: server umount lustre-MDT0001 complete [ 4943.579288] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:24:31 (1784942671) [ 4947.380141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.689204] LustreError: 193041:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.749526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.830330] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4953.057332] LustreError: 193042:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.267801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.450933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4956.507622] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4960.487447] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4963.612799] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4967.212261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.448287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.719377] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4972.590971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4974.422682] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4977.914450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4978.025369] Lustre: lustre-OST0001: new disk, initializing [ 4978.027620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4978.031302] Lustre: 195882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4980.749262] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4982.809070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4982.816019] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4982.870374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4985.848252] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4987.878331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 4987.881016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4987.884817] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 4987.889199] Lustre: Skipped 1 previous similar message [ 4987.895678] Lustre: Skipped 3 previous similar messages [ 4993.223325] Lustre: server umount lustre-OST0000 complete [ 4995.855341] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4995.955148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4996.418500] LustreError: 195392:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.892527] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4997.999601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5001.540221] LustreError: 194862:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.551235] LustreError: 194862:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 5013.477126] LustreError: 194862:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.478311] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5013.478445] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5013.481126] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 5013.481134] Lustre: Skipped 3 previous similar messages [ 5013.486860] LustreError: 194862:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 5013.493207] LustreError: Skipped 1 previous similar message [ 5014.496548] Lustre: server umount lustre-OST0001 complete [ 5030.367129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5030.449216] Lustre: server umount lustre-MDT0000 complete [ 5032.170274] LustreError: 193023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942761 with bad export cookie 11657684985270451358 [ 5032.173991] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.176431] LustreError: 193023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5032.322820] Lustre: server umount lustre-MDT0001 complete [ 5043.719883] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:26:11 (1784942771) [ 5046.739912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.897480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5046.914629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5047.015827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.021249] Lustre: Skipped 1 previous similar message [ 5049.161486] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5053.073793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.124168] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5053.152685] Lustre: 199290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5055.383426] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5058.837945] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5061.611909] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5064.653477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.761098] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5067.660468] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5070.802588] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.929591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5074.002425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.054581] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5076.615497] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5079.810167] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.616267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5082.619715] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 5082.622786] Lustre: Skipped 4 previous similar messages [ 5082.632260] Lustre: Skipped 8 previous similar messages [ 5082.654715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5095.391120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.438192] Lustre: server umount lustre-OST0000 complete [ 5095.748683] LustreError: 201140:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.196111] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5108.197463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5108.202042] Lustre: Skipped 1 previous similar message [ 5108.205678] LustreError: Skipped 1 previous similar message [ 5113.056974] Lustre: server umount lustre-OST0001 complete [ 5118.436450] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 5118.442550] Lustre: Skipped 9 previous similar messages [ 5121.142270] Lustre: server umount lustre-MDT0000 complete [ 5122.934260] LustreError: 199284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942852 with bad export cookie 11657684985270452744 [ 5122.935548] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.941473] LustreError: 199284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5123.128971] Lustre: server umount lustre-MDT0001 complete [ 5138.002413] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:27:45 (1784942865) [ 5160.683755] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5171.628102] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5215.990031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.350376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5216.385844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5216.483388] Lustre: lustre-MDT0000: new disk, initializing [ 5216.579061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.584406] Lustre: Skipped 3 previous similar messages [ 5216.597884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5221.743629] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5232.596645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.772755] Lustre: 207359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5232.824573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5232.830805] Lustre: Skipped 1 previous similar message [ 5232.959888] Lustre: lustre-MDT0001: new disk, initializing [ 5233.118744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5233.145767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5236.989241] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5245.431345] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5250.167223] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5255.194316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5255.413707] Lustre: lustre-OST0000: new disk, initializing [ 5255.417122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5255.421680] Lustre: 209327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5260.909242] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5262.895416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5262.908844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5262.970877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5267.670818] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5270.894796] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5271.074626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5274.513521] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5274.708027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5278.196697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5278.197501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5278.204330] Lustre: Skipped 2 previous similar messages [ 5278.211199] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 5278.221496] Lustre: Skipped 1 previous similar message [ 5282.194050] Lustre: server umount lustre-OST0000 complete [ 5291.265272] Lustre: server umount lustre-MDT0000 complete [ 5293.538000] LustreError: 207369:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.553152] LustreError: 207369:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 9 previous similar messages [ 5293.968774] LustreError: 207354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943023 with bad export cookie 11657684985270454487 [ 5293.977578] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.305893] Lustre: server umount lustre-MDT0001 complete [ 5300.784868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.186421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.195283] Lustre: Skipped 2 previous similar messages [ 5304.949249] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5311.505799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.609583] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5321.433875] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5326.235234] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5331.153297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.205994] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5342.515493] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5346.122918] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5347.356965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5351.357141] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5351.575382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5357.026103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5357.027832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5357.036944] LustreError: Skipped 1 previous similar message [ 5357.061234] Lustre: Skipped 3 previous similar messages [ 5359.250244] Lustre: server umount lustre-OST0000 complete [ 5364.103280] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5367.858539] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5396.024987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5397.314466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.324749] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.328179] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 5397.342847] Lustre: Skipped 1 previous similar message [ 5401.582590] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5407.788557] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5414.272305] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5417.955867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5417.959671] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 5417.968517] LustreError: Skipped 1 previous similar message [ 5417.992410] Lustre: Skipped 5 previous similar messages [ 5419.701229] Lustre: server umount lustre-OST0000 complete [ 5437.407259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5437.569782] Lustre: server umount lustre-MDT0000 complete [ 5438.444348] LustreError: 212284:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.459891] LustreError: 212284:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 5440.636730] LustreError: 211526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943169 with bad export cookie 11657684985270455369 [ 5440.640255] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.649217] LustreError: 211526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5440.977743] Lustre: server umount lustre-MDT0001 complete [ 5448.396518] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:32:55 (1784943175) [ 5454.122650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.705297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.710438] Lustre: Skipped 3 previous similar messages [ 5459.046664] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5466.712478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.058447] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5477.431329] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5482.506859] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5488.678681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.046418] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5502.687750] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5506.544766] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5506.712116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5510.601070] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5510.755096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5516.313401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5516.461940] Lustre: lustre-OST0001: new disk, initializing [ 5516.466975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5516.476393] Lustre: 220886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5521.846686] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5524.045590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5524.063461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5524.130133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5530.524254] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.958619] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5534.136992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5537.957433] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5538.162833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5539.811492] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5539.818179] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5539.830720] Lustre: Skipped 5 previous similar messages [ 5545.882379] Lustre: server umount lustre-OST0001 complete [ 5550.867996] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5551.068548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5554.765343] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5554.968227] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5570.544348] LustreError: 219340:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.557925] LustreError: 219340:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 15 previous similar messages [ 5581.879213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.004242] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5587.442666] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5587.453539] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5587.453682] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 5587.475300] Lustre: Skipped 1 previous similar message [ 5593.058387] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.137520] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5601.663940] Lustre: server umount lustre-OST0001 complete [ 5623.693351] Lustre: server umount lustre-OST0000 complete [ 5632.833154] Lustre: server umount lustre-MDT0000 complete [ 5635.826864] LustreError: 223244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943364 with bad export cookie 11657684985270456328 [ 5635.830970] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.846194] LustreError: 223244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5650.401172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5650.644259] Lustre: server umount lustre-MDT0001 complete [ 5669.426025] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:36:36 (1784943396) [ 5675.770684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.068194] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5676.113911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5679.919217] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5686.607585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.693584] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5686.746688] Lustre: 226127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5690.701790] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5696.962974] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5701.429540] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.827905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.011706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5712.773499] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5718.909921] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5726.784477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.867822] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5727.136866] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5727.141752] Lustre: Skipped 7 previous similar messages [ 5732.531284] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5738.834929] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.310232] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5743.937895] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5744.118678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5747.821953] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5748.049222] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5752.984198] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92324523a800.ost_server_uuid 50 [ 5754.309649] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92324523a800.ost_server_uuid in IDLE state after 0 sec [ 5755.109648] Lustre: Permanently deactivating lustre-OST0001 [ 5755.130898] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5758.247430] Lustre: Permanently reactivating lustre-OST0001 [ 5763.555164] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 5763.561739] LustreError: Skipped 3 previous similar messages [ 5763.567776] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 5763.578283] Lustre: Skipped 19 previous similar messages [ 5767.789634] Lustre: server umount lustre-OST0001 complete [ 5785.055113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5785.182499] Lustre: server umount lustre-OST0000 complete [ 5794.789227] Lustre: server umount lustre-MDT0000 complete [ 5797.815331] LustreError: 226119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943526 with bad export cookie 11657684985270457623 [ 5797.821196] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.835433] LustreError: 226119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5798.151865] Lustre: server umount lustre-MDT0001 complete [ 5818.181617] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:39:05 (1784943545) [ 5825.314468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.587284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5825.616441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5825.627773] Lustre: Skipped 5 previous similar messages [ 5829.465482] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5835.403549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.459964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5835.482831] Lustre: 232820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5838.780117] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5844.306773] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5848.847658] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5853.699692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5853.869457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5853.884834] Lustre: Permanently deactivating lustre-OST0000 [ 5853.898282] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5858.546626] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5861.923647] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5864.678291] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5869.819214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.899389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5874.605950] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5879.434348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5880.083328] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5881.404587] Lustre: Permanently reactivating lustre-OST0000 [ 5881.424922] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5881.430874] Lustre: Skipped 2 previous similar messages [ 5886.005769] Lustre: lustre-OST0000: Received new MDS connection from 192.168.203.114@tcp, remove former export from same NID [ 5886.062724] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 192.168.203.114@tcp) reconnecting [ 5886.073373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.088932] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 5886.100889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5892.083544] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5892.098499] Lustre: Skipped 12 previous similar messages [ 5896.804739] Lustre: server umount lustre-OST0001 complete [ 5900.261957] LustreError: 234667:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.283183] LustreError: 234667:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 31 previous similar messages [ 5905.866195] Lustre: server umount lustre-OST0000 complete [ 5923.295314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5923.423604] Lustre: server umount lustre-MDT0000 complete [ 5926.062864] LustreError: 232813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943655 with bad export cookie 11657684985270459268 [ 5926.068757] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5926.077623] LustreError: 232813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5926.284504] Lustre: server umount lustre-MDT0001 complete [ 5933.440299] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 5935.618792] Key type lgssc unregistered [ 5935.846640] LNet: 237674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.855317] LNetError: 237674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.869795] LNet: Removed LNI 192.168.203.114@tcp [ 5936.557763] Key type .llcrypt unregistered [ 5936.563734] Key type ._llcrypt unregistered [ 5953.740344] Key type ._llcrypt registered [ 5953.742219] Key type .llcrypt registered [ 5953.834310] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5954.525070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5954.563959] alg: No test for adler32 (adler32-zlib) [ 5955.570310] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5955.836691] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 5957.527141] Key type lgssc registered [ 5958.385810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5966.640247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5966.667225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.051721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.681763] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5978.161554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.256965] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5978.274865] Lustre: 239109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5978.282169] Lustre: Permanently deactivating lustre-MDT0001 [ 5978.293402] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5978.504439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5981.436310] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5984.789314] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5986.984896] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5991.407378] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5996.853739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.149482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.137971] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6003.310211] LustreError: 240966:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6003.322163] LustreError: 240966:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 6003.328531] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6007.623438] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.812168] LustreError: 240966:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.030497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6013.259816] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.326156] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6014.333935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6014.338550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6018.196144] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6023.459901] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.732242] Lustre: Permanently reactivating lustre-MDT0001 [ 6026.752007] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6026.761130] Lustre: Skipped 2 previous similar messages [ 6029.359816] Lustre: lustre-MDT0001: Client 6f40794a-a6b2-4c23-9f79-ce9a3ea1ab68 (at 192.168.203.14@tcp) reconnecting [ 6032.992743] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6033.029767] Lustre: lustre-MDT0001: Received new MDS connection from 192.168.203.114@tcp, keep former export from same NID [ 6033.042910] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6033.052232] LustreError: 242832:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6033.053214] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 6033.064631] LustreError: 242832:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6033.064746] LustreError: 242832:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -108 [ 6033.068875] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6033.068875] [ 6033.096728] LustreError: 239113:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6034.550310] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6035.741419] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6040.170166] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6040.355430] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6043.756835] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6043.921315] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6046.080436] Lustre: Permanently deactivating lustre-MDT0001 [ 6046.095446] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6046.101771] Lustre: Skipped 2 previous similar messages [ 6054.494985] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6060.526431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6060.535644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6060.556486] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6063.585313] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6063.591905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6063.600623] Lustre: Skipped 2 previous similar messages [ 6065.012319] Lustre: server umount lustre-MDT0000 complete [ 6067.561560] LustreError: 239102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943796 with bad export cookie 5465066502544639754 [ 6067.568533] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6067.576929] LustreError: 239102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6067.824611] Lustre: server umount lustre-MDT0001 complete [ 6072.977856] Lustre: server umount lustre-OST0000 complete [ 6081.354286] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:43:28 (1784943808) [ 6089.083806] Lustre: server umount lustre-OST0001 complete [ 6093.837334] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6101.006163] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6126.346250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.522258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6126.529070] Lustre: Skipped 2 previous similar messages [ 6126.548103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6126.601459] Lustre: lustre-MDT0000: new disk, initializing [ 6126.647479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.667165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6129.277375] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6136.026520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.108966] Lustre: 247374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6136.158971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6136.163747] Lustre: Skipped 1 previous similar message [ 6136.218323] Lustre: lustre-MDT0001: new disk, initializing [ 6136.271862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.294990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6136.304507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6138.776881] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6145.349927] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6149.155466] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6153.547992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6153.744234] Lustre: lustre-OST0000: new disk, initializing [ 6153.747843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6153.753132] Lustre: 249340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6153.821639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.473571] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6162.967228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6162.974110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6163.029177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6165.654672] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6167.202358] LustreError: 247381:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6173.677239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.794251] Lustre: lustre-OST0001: new disk, initializing [ 6173.797688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6173.802526] Lustre: 250453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6177.239167] LustreError: 247381:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6177.259978] LustreError: 247380:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6178.048863] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6183.472387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6183.485268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6183.573222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6184.670691] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6185.654719] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6187.311750] LustreError: 247380:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6189.031045] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6189.036013] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6189.040435] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 6189.045280] Lustre: Skipped 1 previous similar message [ 6189.048527] Lustre: Skipped 3 previous similar messages [ 6190.550208] Lustre: server umount lustre-OST0001 complete [ 6194.151915] LustreError: 249325:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.167934] LustreError: 249325:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6199.268180] LustreError: 249323:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.276929] LustreError: 249323:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6204.387350] LustreError: 249919:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.394484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6204.401435] LustreError: 249919:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6204.409918] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6204.427439] Lustre: Skipped 3 previous similar messages [ 6209.516544] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6209.525330] Lustre: Skipped 1 previous similar message [ 6210.551534] Lustre: server umount lustre-OST0000 complete [ 6213.089216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6213.095583] LustreError: Skipped 1 previous similar message [ 6213.098223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6213.107464] Lustre: Skipped 1 previous similar message [ 6219.085437] Lustre: server umount lustre-MDT0000 complete [ 6219.748381] LustreError: 247385:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.764177] LustreError: 247385:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6221.498484] LustreError: 247366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943950 with bad export cookie 5465066502544642568 [ 6221.502637] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6221.504729] LustreError: 247366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6227.776473] Lustre: server umount lustre-MDT0001 complete [ 6234.769193] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6236.655536] Key type lgssc unregistered [ 6236.907843] LNet: 252484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6236.916208] LNetError: 252484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6236.929233] LNet: Removed LNI 192.168.203.114@tcp [ 6237.473579] Key type .llcrypt unregistered [ 6237.475618] Key type ._llcrypt unregistered [ 6262.713891] Key type ._llcrypt registered [ 6262.715587] Key type .llcrypt registered [ 6262.791534] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6263.444606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6263.455364] alg: No test for adler32 (adler32-zlib) [ 6264.406743] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6264.578639] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6266.207161] Key type lgssc registered [ 6267.052530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6272.503496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6272.537430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.784384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6273.815824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6273.947952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.927749] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6283.527967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.631795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6283.656752] Lustre: 254672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6283.857863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.864232] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6291.483993] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6295.079934] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6299.769662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.893608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6300.046319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.092749] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6307.912112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6308.927601] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6353.891413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6353.893517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6353.899697] LustreError: Skipped 1 previous similar message [ 6353.911434] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6353.919630] Lustre: Skipped 1 previous similar message [ 6358.949575] Lustre: server umount lustre-OST0000 complete [ 6361.663836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6370.065778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.323816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.443872] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.459711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.463384] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 6374.479384] Lustre: Skipped 1 previous similar message [ 6374.606453] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6379.669763] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6415.334182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6415.334278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6415.337508] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6415.346671] LustreError: Skipped 1 previous similar message [ 6415.366762] Lustre: Skipped 2 previous similar messages [ 6420.455857] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6420.465197] Lustre: Skipped 2 previous similar messages [ 6421.471291] Lustre: server umount lustre-OST0000 complete [ 6425.571434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6425.578179] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6425.582124] Lustre: Skipped 1 previous similar message [ 6430.689668] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6430.694289] Lustre: Skipped 2 previous similar messages [ 6438.367147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.474991] Lustre: server umount lustre-MDT0000 complete [ 6440.820587] LustreError: 254663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944169 with bad export cookie 5005712147754521654 [ 6440.823365] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.829169] LustreError: 254663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6441.050714] Lustre: server umount lustre-MDT0001 complete [ 6447.411467] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6449.048262] Key type lgssc unregistered [ 6449.245767] LNet: 260085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6449.253459] LNetError: 260085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6449.265969] LNet: Removed LNI 192.168.203.114@tcp [ 6449.835745] Key type .llcrypt unregistered [ 6449.837681] Key type ._llcrypt unregistered [ 6464.149879] Key type ._llcrypt registered [ 6464.152433] Key type .llcrypt registered [ 6464.212041] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6464.832164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6464.848680] alg: No test for adler32 (adler32-zlib) [ 6465.771437] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6465.912311] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6467.543226] Key type lgssc registered [ 6468.281562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6473.014902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6473.029879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.268816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.519147] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6481.569629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.805285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.509831] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6489.460452] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6493.052610] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6497.130463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.350623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.442318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6500.971680] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6505.790418] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6522.340822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6522.341438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6522.351065] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6522.362684] Lustre: Skipped 1 previous similar message [ 6527.425309] Lustre: server umount lustre-OST0000 complete [ 6532.578184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6532.584528] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6532.588366] Lustre: Skipped 2 previous similar messages [ 6535.690800] Lustre: server umount lustre-MDT0000 complete [ 6537.699122] LustreError: 261482:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.715556] LustreError: 261482:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6537.824205] LustreError: 261461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944266 with bad export cookie 9824237454967100171 [ 6537.826583] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.831968] LustreError: 261461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6537.999707] Lustre: server umount lustre-MDT0001 complete [ 6543.816400] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6545.455522] Key type lgssc unregistered [ 6545.657520] LNet: 265781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6545.666922] LNetError: 265781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6545.686258] LNet: Removed LNI 192.168.203.114@tcp [ 6546.217180] Key type .llcrypt unregistered [ 6546.219055] Key type ._llcrypt unregistered [ 6557.346321] Key type ._llcrypt registered [ 6557.350216] Key type .llcrypt registered [ 6557.426667] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6557.922404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6557.939020] alg: No test for adler32 (adler32-zlib) [ 6558.909289] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6559.066914] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6560.703155] Key type lgssc registered [ 6561.481645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6566.368193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6566.380402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.627972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.928488] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6574.542011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.773627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.021894] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6581.075332] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.138716] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6588.422650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.647830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.602323] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6592.755717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6597.012840] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6608.356783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6608.362370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6608.375312] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6609.382094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6609.383099] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6609.383654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6609.399466] Lustre: Skipped 1 previous similar message [ 6613.953467] Lustre: server umount lustre-OST0000 complete [ 6619.617548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6619.620487] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6619.627819] Lustre: Skipped 1 previous similar message [ 6622.400404] Lustre: server umount lustre-MDT0000 complete [ 6624.460738] LustreError: 266963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944353 with bad export cookie 16886944705839099630 [ 6624.464238] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6624.466679] LustreError: 266963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6624.644838] Lustre: server umount lustre-MDT0001 complete [ 6630.417209] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6632.088832] Key type lgssc unregistered [ 6632.283838] LNet: 270654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6632.293764] LNetError: 270654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6632.310502] LNet: Removed LNI 192.168.203.114@tcp [ 6632.826123] Key type .llcrypt unregistered [ 6632.828083] Key type ._llcrypt unregistered [ 6647.159161] Key type ._llcrypt registered [ 6647.161844] Key type .llcrypt registered [ 6647.225930] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6647.844135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6647.858053] alg: No test for adler32 (adler32-zlib) [ 6648.806468] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6648.943356] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6650.559163] Key type lgssc registered [ 6651.161590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6656.008230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6656.022814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.321276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6659.821773] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6665.457149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.731302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6668.705587] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6673.808650] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.981908] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.197387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.400898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.038080] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6690.227985] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6702.906436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6723.558493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6723.564478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6723.567458] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6723.581047] Lustre: Skipped 1 previous similar message [ 6728.539671] Lustre: server umount lustre-OST0000 complete [ 6732.257284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6732.264169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6732.278849] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6732.285538] Lustre: Skipped 1 previous similar message [ 6733.793084] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6733.797226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6737.371744] Lustre: server umount lustre-MDT0000 complete [ 6738.915068] LustreError: 272048:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.928429] LustreError: 272048:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6739.536058] LustreError: 272029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944468 with bad export cookie 8847731679974451149 [ 6739.541092] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.548467] LustreError: 272029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.775684] Lustre: server umount lustre-MDT0001 complete [ 6746.358542] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6748.255276] Key type lgssc unregistered [ 6748.468367] LNet: 276372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6748.473481] LNetError: 276372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6749.543429] LNet: Removed LNI 192.168.203.114@tcp [ 6750.165749] Key type .llcrypt unregistered [ 6750.167506] Key type ._llcrypt unregistered [ 6761.807630] Key type ._llcrypt registered [ 6761.809786] Key type .llcrypt registered [ 6761.878462] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6762.442013] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6762.466380] alg: No test for adler32 (adler32-zlib) [ 6763.454360] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6763.643767] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6765.287202] Key type lgssc registered [ 6766.119551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6770.962644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6770.976676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.266699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6775.024275] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6780.468062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.678325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6783.280695] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6788.498297] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6792.567753] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6797.111678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.303508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.419183] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6805.840724] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6817.783419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6828.008114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6828.011312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6828.016141] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6828.026966] Lustre: Skipped 1 previous similar message [ 6833.127500] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6833.133424] Lustre: Skipped 1 previous similar message [ 6833.374179] Lustre: server umount lustre-OST0000 complete [ 6837.223188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6837.232834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6837.243110] Lustre: Skipped 1 previous similar message [ 6837.253130] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6838.244144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6841.829649] Lustre: server umount lustre-MDT0000 complete [ 6843.365390] LustreError: 277589:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.382362] LustreError: 277589:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6844.360323] LustreError: 277554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944573 with bad export cookie 6875329786818174212 [ 6844.369698] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6844.578844] Lustre: server umount lustre-MDT0001 complete [ 6850.944906] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6852.743041] Key type lgssc unregistered [ 6852.942727] LNet: 281308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6852.951051] LNetError: 281308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6852.969382] LNet: Removed LNI 192.168.203.114@tcp [ 6853.545886] Key type .llcrypt unregistered [ 6853.548859] Key type ._llcrypt unregistered [ 6860.259671] Key type ._llcrypt registered [ 6860.262442] Key type .llcrypt registered [ 6860.333953] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6870.857504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6879.623193] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6880.232617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.241033] alg: No test for adler32 (adler32-zlib) [ 6881.175229] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6881.306673] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6882.927156] Key type lgssc registered [ 6883.619155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6904.857833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6904.885228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.019716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6906.037384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6906.090364] Lustre: lustre-MDT0000: new disk, initializing [ 6906.144175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.155846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6908.591142] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6915.122446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.190672] Lustre: 284836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6915.236873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6915.240620] Lustre: Skipped 1 previous similar message [ 6915.293291] Lustre: lustre-MDT0001: new disk, initializing [ 6915.359368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6915.383996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6915.398208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6918.032710] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6924.350971] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6928.036599] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6931.929152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.081202] Lustre: lustre-OST0000: new disk, initializing [ 6932.084711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6932.089189] Lustre: 286801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6932.149707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.652279] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6940.688356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.693468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.755990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6940.872226] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.941977] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6943.043881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6944.934340] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6945.036137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6950.884756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 6950.884849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6950.895025] LustreError: Skipped 1 previous similar message [ 6950.906658] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6950.911879] Lustre: Skipped 1 previous similar message [ 6952.173996] Lustre: server umount lustre-OST0000 complete [ 6956.001658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6956.015100] Lustre: Skipped 1 previous similar message [ 6956.022228] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 6956.027483] Lustre: Skipped 1 previous similar message [ 6960.335205] Lustre: server umount lustre-MDT0000 complete [ 6962.381594] LustreError: 284827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944691 with bad export cookie 2670120184448325332 [ 6962.381911] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6962.389698] LustreError: 284827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6962.567413] Lustre: server umount lustre-MDT0001 complete [ 6966.998600] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:58:14 (1784944694) [ 6970.699242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.962667] LustreError: 289212:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6970.977396] LustreError: 289212:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6971.004052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.333354] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6976.486132] LustreError: 289213:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.064742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.287555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.565092] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6984.640903] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6987.624670] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.288729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.479378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.564833] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6998.393828] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7009.250160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7009.255305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7009.256760] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7009.261955] Lustre: Skipped 2 previous similar messages [ 7010.915484] Lustre: server umount lustre-OST0000 complete [ 7014.372987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7014.373141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7014.386398] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7014.392625] Lustre: Skipped 1 previous similar message [ 7014.396323] Lustre: Skipped 1 previous similar message [ 7018.958281] Lustre: server umount lustre-MDT0000 complete [ 7019.490407] LustreError: 289209:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.507751] LustreError: 289209:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7021.069510] LustreError: 291051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944750 with bad export cookie 2670120184448326214 [ 7021.073783] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7021.079556] LustreError: 291051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7021.252672] Lustre: server umount lustre-MDT0001 complete [ 7027.286096] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7028.989210] Key type lgssc unregistered [ 7029.169484] LNet: 292740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7029.174597] LNetError: 292740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7029.191045] LNet: Removed LNI 192.168.203.114@tcp [ 7029.637552] Key type .llcrypt unregistered [ 7029.639905] Key type ._llcrypt unregistered [ 7043.011922] Key type ._llcrypt registered [ 7043.015092] Key type .llcrypt registered [ 7043.082720] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7051.367394] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7051.883222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7051.913083] alg: No test for adler32 (adler32-zlib) [ 7052.839465] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7052.970086] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7054.575645] Key type lgssc registered [ 7055.138300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7078.922185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7078.942344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.085270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7080.105148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7080.148238] Lustre: lustre-MDT0000: new disk, initializing [ 7080.194929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.211120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7082.253468] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7088.034166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.098980] Lustre: 296541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7088.132695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7088.135980] Lustre: Skipped 1 previous similar message [ 7088.178946] Lustre: lustre-MDT0001: new disk, initializing [ 7088.226693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.250033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7088.255282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7090.114931] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7095.295621] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.304825] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7101.852188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7102.014483] Lustre: lustre-OST2710: new disk, initializing [ 7102.018376] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7102.023503] Lustre: 298508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7102.079312] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7104.880178] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7109.831854] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7111.188139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7111.194111] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7111.265985] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7113.877942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.980555] Lustre: lustre-OST03e8: new disk, initializing [ 7113.985506] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7113.989379] Lustre: 299571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7114.045894] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7117.150924] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7120.917094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7120.927610] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7121.007134] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7121.274465] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7125.396767] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7125.536451] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7128.106312] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7128.258482] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7130.835857] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7130.969771] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7133.518309] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7133.649808] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7136.232742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7136.238881] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7136.240227] Lustre: Skipped 2 previous similar messages [ 7136.250311] Lustre: Skipped 3 previous similar messages [ 7141.351803] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7141.356550] Lustre: Skipped 3 previous similar messages [ 7141.989912] Lustre: server umount lustre-MDT0000 complete [ 7144.024991] LustreError: 296534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944873 with bad export cookie 13112463697548087428 [ 7144.026610] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.038221] LustreError: 296534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7144.209989] Lustre: server umount lustre-MDT0001 complete [ 7156.688338] Lustre: server umount lustre-OST2710 complete [ 7168.428162] Lustre: server umount lustre-OST03e8 complete [ 7175.666660] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7179.914512] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7197.627306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.753302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7197.771269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7197.824286] Lustre: lustre-MDT0000: new disk, initializing [ 7197.873707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.884427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7200.009504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7206.063760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.134135] Lustre: 304822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7206.164418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7206.167258] Lustre: Skipped 1 previous similar message [ 7206.212234] Lustre: lustre-MDT0001: new disk, initializing [ 7206.264441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.286246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7206.290869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7208.268088] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7213.671174] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7216.567154] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7219.823903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.947832] Lustre: 306791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7222.667396] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7227.368985] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7229.648352] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7229.967203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7229.972281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7230.024701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7230.807832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7232.959284] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7233.078393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7235.045427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7235.045475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7235.053454] Lustre: Skipped 1 previous similar message [ 7235.059782] LustreError: Skipped 1 previous similar message [ 7235.060753] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7240.170913] Lustre: server umount lustre-OST0000 complete [ 7242.209058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7242.216229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7242.225887] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7242.232128] Lustre: Skipped 1 previous similar message [ 7245.284551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7248.368881] Lustre: server umount lustre-MDT0000 complete [ 7250.063217] LustreError: 304816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944979 with bad export cookie 13112463697548089094 [ 7250.065238] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.071613] LustreError: 304816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7250.225115] Lustre: server umount lustre-MDT0001 complete [ 7254.555083] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:03:02 (1784944982) [ 7255.370949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7256.242880] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:03:04 (1784944984) [ 7257.951391] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7270.417792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.558236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7270.570827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7270.647418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.651892] Lustre: Skipped 1 previous similar message [ 7272.669189] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7276.616438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.677556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7276.692166] Lustre: 310651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7278.560073] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7295.529027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7295.639977] Lustre: Denying initial registration attempt from nid 192.168.203.114@tcp, specified as failover [ 7295.644247] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7295.648132] LustreError: 312102:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 7295.651109] LustreError: 312102:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7295.653825] LustreError: 312102:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7295.684822] Lustre: server umount lustre-OST0000 complete [ 7295.686872] LustreError: 312102:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7300.579266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7300.580686] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7300.586888] Lustre: Skipped 1 previous similar message [ 7300.590701] Lustre: Skipped 2 previous similar messages [ 7304.602061] Lustre: server umount lustre-MDT0000 complete [ 7306.249175] LustreError: 310641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945035 with bad export cookie 13112463697548089976 [ 7306.251601] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.256451] LustreError: 310641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7306.417092] Lustre: server umount lustre-MDT0001 complete [ 7310.997722] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7312.279690] Key type lgssc unregistered [ 7312.432948] LNet: 313219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7312.437236] LNetError: 313219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7312.453958] LNet: Removed LNI 192.168.203.114@tcp [ 7312.848454] Key type .llcrypt unregistered [ 7312.850582] Key type ._llcrypt unregistered [ 7317.664609] Key type ._llcrypt registered [ 7317.667866] Key type .llcrypt registered [ 7317.733119] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7318.156824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7318.210198] alg: No test for adler32 (adler32-zlib) [ 7319.126647] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7319.254685] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7320.855393] Key type lgssc registered [ 7321.346400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7350.239862] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7354.254494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7354.267246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.409878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7355.420206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7355.497642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.426708] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7361.335368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.390116] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7361.405197] Lustre: 315960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7361.532327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.529979] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7367.263128] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7369.982427] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7386.276654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7386.387067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7386.485101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.984923] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7391.980303] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7392.882644] Lustre: server umount lustre-OST0000 complete [ 7396.356420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7396.358961] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7396.363381] Lustre: Skipped 1 previous similar message [ 7396.368857] Lustre: Skipped 1 previous similar message [ 7400.755578] Lustre: server umount lustre-MDT0000 complete [ 7402.681637] LustreError: 315950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945131 with bad export cookie 7095557441722665745 [ 7402.683955] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7402.689834] LustreError: 315950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7402.862738] Lustre: server umount lustre-MDT0001 complete [ 7407.785409] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7409.124769] Key type lgssc unregistered [ 7409.273489] LNet: 319524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7409.280663] LNetError: 319524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7409.288899] LNet: Removed LNI 192.168.203.114@tcp [ 7409.708525] Key type .llcrypt unregistered [ 7409.710712] Key type ._llcrypt unregistered [ 7421.098025] Key type ._llcrypt registered [ 7421.099591] Key type .llcrypt registered [ 7421.151080] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7421.608559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7421.655733] alg: No test for adler32 (adler32-zlib) [ 7422.553561] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7422.695483] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7424.327391] Key type lgssc registered [ 7424.829250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7428.401815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7428.413274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.637767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.631758] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7435.625957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.809463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7437.660923] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7440.720695] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.099496] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7448.168368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7448.342758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.805367] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7453.786456] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7470.414146] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 7471.078869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7471.080869] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7471.089611] Lustre: Skipped 2 previous similar messages [ 7471.096806] Lustre: Skipped 2 previous similar messages [ 7474.927974] Lustre: server umount lustre-MDT0000 complete [ 7475.520244] LustreError: 322019:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.529072] LustreError: 322019:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 7476.196225] LustreError: 320917:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.203820] LustreError: 320917:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 7476.487506] LustreError: 320899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945205 with bad export cookie 6371806707106464131 [ 7476.490545] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7476.495992] LustreError: 320899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7480.643651] LustreError: 323373:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.660728] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 7481.318428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7482.710731] Lustre: server umount lustre-MDT0001 complete [ 7485.985141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7489.749094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.796321] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x586d3470da885d83 [ 7501.804708] Lustre: MGC192.168.203.114@tcp: Connection restored to 0@lo (at 0@lo) [ 7501.959836] LustreError: 322746:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.970633] LustreError: 322746:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7502.000215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7502.031729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7504.102545] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7507.429430] LustreError: 322747:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.314040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.512731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7510.525122] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7511.589693] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 7514.320468] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7518.306983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7521.033491] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7521.790529] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7521.828439] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 7523.813836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7523.817769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7523.827394] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7523.832036] Lustre: Skipped 1 previous similar message [ 7526.888284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7532.003912] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7532.011252] Lustre: Skipped 3 previous similar messages [ 7538.143149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7538.221085] Lustre: server umount lustre-OST0000 complete [ 7542.242564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7542.249466] Lustre: Skipped 1 previous similar message [ 7542.249848] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7542.259752] Lustre: Skipped 3 previous similar messages [ 7545.974529] Lustre: server umount lustre-MDT0000 complete [ 7547.362133] LustreError: 324534:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.373327] LustreError: 324534:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7547.875561] LustreError: 320898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945277 with bad export cookie 6371806707106471299 [ 7547.879216] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.888950] LustreError: 320898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7548.142200] Lustre: server umount lustre-MDT0001 complete [ 7552.974836] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7554.329899] Key type lgssc unregistered [ 7554.505736] LNet: 327202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7554.509959] LNetError: 327202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7554.521984] LNet: Removed LNI 192.168.203.114@tcp [ 7554.972938] Key type .llcrypt unregistered [ 7554.974615] Key type ._llcrypt unregistered [ 7567.526751] Key type ._llcrypt registered [ 7567.528674] Key type .llcrypt registered [ 7567.599989] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7568.196369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.232675] alg: No test for adler32 (adler32-zlib) [ 7569.129128] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7569.251483] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7570.847419] Key type lgssc registered [ 7571.535659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7575.493497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.504103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.728215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.718315] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7582.918286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.128828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7585.052131] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7588.766706] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7591.502566] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7593.442572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7593.448560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7593.456278] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7594.467665] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7594.468472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7599.585641] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7599.590444] Lustre: Skipped 2 previous similar messages [ 7604.704824] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7604.711912] Lustre: Skipped 1 previous similar message [ 7607.263191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7607.359867] Lustre: server umount lustre-MDT0000 complete [ 7609.158791] LustreError: 328576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945338 with bad export cookie 4765193253699680046 [ 7609.165994] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7609.169953] LustreError: 328576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7609.320780] Lustre: server umount lustre-MDT0001 complete [ 7616.112417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.273427] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7616.291510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7616.410075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.333193] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7622.379072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.439984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7622.459630] Lustre: 331180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7622.606938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.566147] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7628.025877] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7630.508350] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7636.071462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.175379] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7636.185779] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7636.195270] LustreError: 333065:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7636.203514] LustreError: 333065:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7636.210580] LustreError: 333065:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7636.254474] Lustre: server umount lustre-OST0000 complete [ 7636.257079] LustreError: 333065:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7642.346218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.472566] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7642.563798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.123340] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7648.522666] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7651.885965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7652.283932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.326126] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7652.367876] Lustre: lustre-OST0001: new disk, initializing [ 7652.370859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7652.375032] Lustre: 334603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7652.420891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7655.066350] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7658.517954] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7659.547992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7659.552819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7659.597598] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7664.613608] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7664.615386] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7664.620917] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 7664.623861] LustreError: Skipped 1 previous similar message [ 7664.626170] Lustre: Skipped 1 previous similar message [ 7665.932425] Lustre: server umount lustre-OST0001 complete [ 7669.732735] LustreError: 333532:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.733859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7669.741868] LustreError: 333532:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 7669.749274] Lustre: Skipped 2 previous similar messages [ 7673.827212] Lustre: server umount lustre-OST0000 complete [ 7678.946868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7678.952156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7678.966660] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7678.969619] Lustre: Skipped 2 previous similar messages [ 7681.722576] Lustre: server umount lustre-MDT0000 complete [ 7683.682390] LustreError: 333546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945412 with bad export cookie 4765193253699680585 [ 7683.686516] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7683.687920] LustreError: 333546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7683.884749] Lustre: server umount lustre-MDT0001 complete [ 7688.770683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7690.067736] Key type lgssc unregistered [ 7690.211551] LNet: 336559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7690.218422] LNetError: 336559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7690.230628] LNet: Removed LNI 192.168.203.114@tcp [ 7690.623341] Key type .llcrypt unregistered [ 7690.624919] Key type ._llcrypt unregistered [ 7729.936025] Key type ._llcrypt registered [ 7729.937377] Key type .llcrypt registered [ 7729.982342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7736.841217] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7737.186304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.196323] alg: No test for adler32 (adler32-zlib) [ 7738.071246] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7738.183543] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7739.775272] Key type lgssc registered [ 7740.250218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.239815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.259813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.355952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7755.371511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7755.411563] Lustre: lustre-MDT0000: new disk, initializing [ 7755.454349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.463742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7757.177117] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7762.164551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.218187] Lustre: 341472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7762.248801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7762.252126] Lustre: Skipped 1 previous similar message [ 7762.292736] Lustre: lustre-MDT0001: new disk, initializing [ 7762.335498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.350222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7762.355635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7763.929382] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7768.263232] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7770.612543] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7773.612331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.758329] Lustre: lustre-OST0000: new disk, initializing [ 7773.758484] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (not set up) [ 7773.761678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7773.769488] Lustre: 343442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7773.803295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.137271] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7779.327805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7779.334188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7779.357627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7781.028885] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7783.121856] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7783.224313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7785.480548] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7785.598549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7788.512921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7788.519157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7788.522131] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7788.526405] Lustre: Skipped 1 previous similar message [ 7792.809806] Lustre: server umount lustre-OST0000 complete [ 7798.242410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7798.246651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7798.254237] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7798.257966] Lustre: Skipped 1 previous similar message [ 7800.682742] Lustre: server umount lustre-MDT0000 complete [ 7802.310727] LustreError: 341467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945531 with bad export cookie 12194100873360380626 [ 7802.314129] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7802.315177] LustreError: 341467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7802.453834] Lustre: server umount lustre-MDT0001 complete [ 7805.963155] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:12:13 (1784945533) [ 7809.132361] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:12:17 (1784945537) [ 7812.694183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.939484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.585670] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7818.209835] LustreError: 346191:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.215868] LustreError: 346191:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 7818.560792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.753095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7820.821235] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7824.317203] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7827.031387] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7830.037043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.202948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.624155] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7835.800420] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.630667] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7851.380965] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7852.163717] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7852.907718] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7853.735925] Lustre: Failing over lustre-MDT0000 [ 7853.908753] Lustre: server umount lustre-MDT0000 complete [ 7854.050525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7854.054149] LustreError: 346187:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.058135] Lustre: Skipped 3 previous similar messages [ 7854.067869] LustreError: 346187:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7858.515763] LustreError: 346188:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7858.523387] LustreError: 346188:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7863.618149] LustreError: 346217:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.626153] LustreError: 346217:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 7867.609249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.680605] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.784360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.808415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7868.738540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7869.374293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7873.003032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 7873.010081] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7873.028524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [ 7875.044148] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7875.636930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7877.669178] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7878.454335] Lustre: Failing over lustre-MDT0000 [ 7878.651530] Lustre: server umount lustre-MDT0000 complete [ 7878.991187] LustreError: 346187:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.001100] LustreError: 346187:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 7883.233816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7883.234289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7883.242071] Lustre: Skipped 1 previous similar message [ 7888.352761] LustreError: 346210:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.359979] LustreError: 346210:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 7892.134027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.191848] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.326266] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7893.818329] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7894.337165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7897.577328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp) [ 7897.581362] Lustre: Skipped 2 previous similar messages [ 7897.595789] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7897.616247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:97) [ 7899.771649] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7900.399601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7907.814582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7907.816189] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7907.821687] Lustre: Skipped 2 previous similar messages [ 7907.822028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7907.824701] Lustre: Skipped 2 previous similar messages [ 7908.892207] Lustre: server umount lustre-MDT0000 complete [ 7910.381313] LustreError: 346171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945639 with bad export cookie 12194100873360383307 [ 7910.383443] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.387870] LustreError: 346171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7910.528475] Lustre: server umount lustre-MDT0001 complete [ 7922.466889] Lustre: server umount lustre-OST0000 complete [ 7928.303850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.522560] LustreError: 352539:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.532567] LustreError: 352539:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7928.563391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.567605] Lustre: Skipped 1 previous similar message [ 7930.181780] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7933.467109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.153569] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7937.935201] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.012896] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7942.481798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7944.808176] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7947.525288] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7952.878646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:129) [ 7957.987184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7957.987738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 7957.989199] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 7957.991965] Lustre: Skipped 2 previous similar messages [ 7960.466206] Lustre: server umount lustre-OST0000 complete [ 7967.909528] Lustre: server umount lustre-MDT0000 complete [ 7968.224262] LustreError: 353267:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.232224] LustreError: 353267:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 7969.375287] LustreError: 352520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945698 with bad export cookie 12194100873360384028 [ 7969.379430] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7969.381881] LustreError: 352520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7969.514081] Lustre: server umount lustre-MDT0001 complete [ 7973.472138] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7974.603568] Key type lgssc unregistered [ 7974.742505] LNet: 356069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7974.745761] LNetError: 356069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7974.753308] LNet: Removed LNI 192.168.203.114@tcp [ 7975.100163] Key type .llcrypt unregistered [ 7975.101714] Key type ._llcrypt unregistered [ 7985.544014] Key type ._llcrypt registered [ 7985.545358] Key type .llcrypt registered [ 7985.586905] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7985.915896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7985.948072] alg: No test for adler32 (adler32-zlib) [ 7986.812915] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 7986.909148] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7988.495161] Key type lgssc registered [ 7988.957922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7991.950745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7991.958897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.120631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.570804] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7997.679072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.820713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.351216] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8001.920847] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8003.854615] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.138378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.260753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.314745] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8010.886481] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8018.535384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:161) [ 8028.640773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8028.644101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8028.650108] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8029.155944] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8029.156473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8031.456157] Lustre: server umount lustre-MDT0000 complete [ 8032.813343] LustreError: 357587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945761 with bad export cookie 8374883150794141629 [ 8032.815300] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8032.818205] LustreError: 357587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8032.934668] Lustre: server umount lustre-MDT0001 complete [ 8044.715656] Lustre: server umount lustre-OST0000 complete [ 8050.069512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.266333] LustreError: 361291:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.277261] LustreError: 361291:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8050.297178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.908937] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8055.376662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.540380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8057.215480] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8060.148623] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8062.278273] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8065.004465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.157539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.573926] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8070.469146] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8075.437942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 8085.985840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8085.987590] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8085.992419] Lustre: Skipped 2 previous similar messages [ 8085.997298] Lustre: Skipped 2 previous similar messages [ 8088.983057] Lustre: server umount lustre-MDT0000 complete [ 8090.479270] LustreError: 361271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945819 with bad export cookie 8374883150794142728 [ 8090.480915] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8090.483844] LustreError: 361271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8090.596212] Lustre: server umount lustre-MDT0001 complete [ 8102.371923] Lustre: server umount lustre-OST0000 complete [ 8108.314634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.508318] LustreError: 365197:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8108.536590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8110.133387] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8113.540466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.633492] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (not set up) [ 8113.636962] Lustre: Skipped 1 previous similar message [ 8115.223463] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8117.984350] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8119.903666] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8122.416393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8124.734116] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8127.230772] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8134.819200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:225) [ 8139.233712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8139.239767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8139.246762] Lustre: Skipped 1 previous similar message [ 8139.249756] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8140.259249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8140.946369] Lustre: server umount lustre-OST0000 complete [ 8144.352279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8144.356310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8144.361869] Lustre: Skipped 1 previous similar message [ 8144.479372] Lustre: server umount lustre-MDT0000 complete [ 8145.376572] LustreError: 366546:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8145.381474] LustreError: 366546:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8145.810346] LustreError: 365177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945874 with bad export cookie 8374883150794143729 [ 8145.813471] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8145.815324] LustreError: 365177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8145.922725] Lustre: server umount lustre-MDT0001 complete [ 8149.879883] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8150.979852] Key type lgssc unregistered [ 8151.110485] LNet: 368746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8151.114432] LNetError: 368746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8151.123361] LNet: Removed LNI 192.168.203.114@tcp [ 8151.464838] Key type .llcrypt unregistered [ 8151.466319] Key type ._llcrypt unregistered [ 8161.592716] Key type ._llcrypt registered [ 8161.594049] Key type .llcrypt registered [ 8161.635168] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8162.087462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8162.095202] alg: No test for adler32 (adler32-zlib) [ 8162.972246] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8163.078110] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8164.671164] Key type lgssc registered [ 8165.118559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8168.027656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8168.036996] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.041463] LustreError: 370165:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8168.052333] LustreError: 370165:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8168.056386] LustreError: 370165:0:(obd_mount.c:257:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8168.060134] LustreError: 370165:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8168.064512] LustreError: 370165:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8170.840078] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8170.843892] LustreError: 370561:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8170.853249] LustreError: 370561:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8170.856871] LustreError: 370561:0:(obd_mount.c:257:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8170.860689] LustreError: 370561:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8170.864441] LustreError: 370561:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8177.781910] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8178.925379] Key type lgssc unregistered [ 8179.068346] LNet: 371562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8179.071330] LNetError: 371562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8179.079267] LNet: Removed LNI 192.168.203.114@tcp [ 8179.390445] Key type .llcrypt unregistered [ 8179.391735] Key type ._llcrypt unregistered [ 8185.883416] Key type ._llcrypt registered [ 8185.884690] Key type .llcrypt registered [ 8185.924185] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8192.458456] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8192.788599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8192.798603] alg: No test for adler32 (adler32-zlib) [ 8193.663628] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8193.760667] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8195.351167] Key type lgssc registered [ 8195.759306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8208.361760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8208.371271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.469367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8209.483052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8209.525420] Lustre: lustre-MDT0000: new disk, initializing [ 8209.552612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.562482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8211.053043] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8215.401638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.448049] Lustre: 374692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8215.476117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8215.479233] Lustre: Skipped 1 previous similar message [ 8215.518516] Lustre: lustre-MDT0001: new disk, initializing [ 8215.557127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8215.574057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8215.579046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8217.047565] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8220.880123] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8222.768845] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8225.093842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8225.197145] Lustre: lustre-OST0000: new disk, initializing [ 8225.199816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8225.203204] Lustre: 376664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8225.234233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8227.256154] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8231.154310] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8232.633697] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8233.983528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8233.987737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8234.024086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8234.752230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8236.269329] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8236.358960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.076538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8239.077114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8239.080333] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8239.081064] LustreError: Skipped 1 previous similar message [ 8239.086753] Lustre: Skipped 1 previous similar message [ 8243.238200] Lustre: server umount lustre-OST0000 complete [ 8246.240781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8246.244453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8246.251375] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8246.253715] Lustre: Skipped 1 previous similar message [ 8249.313132] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8249.313639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8250.777364] Lustre: server umount lustre-MDT0000 complete [ 8252.074221] LustreError: 374685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945981 with bad export cookie 8385359963001868124 [ 8252.077033] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.079090] LustreError: 374685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8252.214825] Lustre: server umount lustre-MDT0001 complete [ 8255.276519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8255.915146] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:19:43 (1784945983) [ 8258.330618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.532429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.015950] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8263.073573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.226492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8264.725785] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8267.437826] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8269.432910] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8271.815450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8271.943880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.061608] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8276.757297] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8279.377559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8279.447419] Lustre: lustre-OST0001: new disk, initializing [ 8279.450129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8279.454714] Lustre: 381965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8281.290306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8281.295790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8281.326954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8281.817236] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8285.949176] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8289.765061] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8289.766061] Lustre: lustre-OST0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 8289.770619] Lustre: Skipped 1 previous similar message [ 8289.773397] Lustre: Skipped 1 previous similar message [ 8293.953449] Lustre: server umount lustre-OST0001 complete [ 8294.884671] LustreError: 380951:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.892704] LustreError: 380951:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 8300.005971] LustreError: 380951:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.013988] LustreError: 380951:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8303.012776] LustreError: 382508:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.020304] LustreError: 382508:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8308.195711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8308.196515] LustreError: 380953:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.197524] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8308.197529] Lustre: Skipped 2 previous similar messages [ 8308.201776] Lustre: Skipped 1 previous similar message [ 8308.209390] LustreError: 380953:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8312.275397] Lustre: server umount lustre-OST0000 complete [ 8314.336506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8318.433455] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8318.435194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8318.436101] Lustre: Skipped 2 previous similar messages [ 8318.439975] Lustre: Skipped 1 previous similar message [ 8328.159114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8328.227596] Lustre: server umount lustre-MDT0000 complete [ 8328.672388] LustreError: 379132:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.678729] LustreError: 379132:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8329.429078] LustreError: 379158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946058 with bad export cookie 8385359963001868992 [ 8329.432304] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.433121] LustreError: 379158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8335.590821] Lustre: server umount lustre-MDT0001 complete [ 8338.998575] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8339.998033] Key type lgssc unregistered [ 8340.124341] LNet: 383949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8340.127555] LNetError: 383949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8340.137281] LNet: Removed LNI 192.168.203.114@tcp [ 8340.442074] Key type .llcrypt unregistered [ 8340.443419] Key type ._llcrypt unregistered [ 8358.341201] Key type ._llcrypt registered [ 8358.342593] Key type .llcrypt registered [ 8358.378052] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8358.770889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8358.801667] alg: No test for adler32 (adler32-zlib) [ 8359.652738] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8359.737267] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8361.311176] Key type lgssc registered [ 8361.645420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8364.038992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8364.044591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.120954] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8365.127720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8365.179138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.414404] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8369.218854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.251675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8369.263100] Lustre: 386723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8369.373541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8370.771427] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8373.229060] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.041752] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8376.801160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8376.801881] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8376.806938] Lustre: Skipped 1 previous similar message [ 8381.921421] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8381.924593] Lustre: Skipped 2 previous similar messages [ 8387.041446] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8387.044185] Lustre: Skipped 1 previous similar message [ 8390.111258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8390.171655] Lustre: server umount lustre-MDT0000 complete [ 8391.516309] LustreError: 386715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946120 with bad export cookie 16236584737325449472 [ 8391.519500] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8391.521858] LustreError: 386715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8391.636040] Lustre: server umount lustre-MDT0001 complete [ 8395.124650] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:22:03 (1784946123) [ 8397.478700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8397.661501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.119902] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8402.278922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.417022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8403.962504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8406.803772] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8408.991973] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8411.522506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.595819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8411.637164] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (not set up) [ 8411.683680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.791527] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8416.430272] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8425.239254] Lustre: Permanently deactivating lustre-OST0000 [ 8425.244960] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8425.303620] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8425.864043] LustreError: 392027:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8425.867575] LustreError: 392027:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8425.870398] LustreError: 392027:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8426.498832] LustreError: 392076:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8426.501990] LustreError: 392076:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8433.232121] Lustre: server umount lustre-OST0000 complete [ 8434.142343] LustreError: 392325:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8434.145349] LustreError: 392325:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8437.217316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8437.218085] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8437.222646] Lustre: Skipped 1 previous similar message [ 8444.405965] Lustre: server umount lustre-MDT0000 complete [ 8445.657625] LustreError: 391924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946174 with bad export cookie 16236584737325450039 [ 8445.661509] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.664366] LustreError: 391924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8455.180346] Lustre: server umount lustre-MDT0001 complete [ 8458.222288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.729802] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8460.738782] LustreError: 393483:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8462.945493] Lustre: 393821:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.114@tcp [ 8463.709189] Lustre: server umount MGS complete [ 8467.838708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.010378] LustreError: 394544:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.015442] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8468.018271] LustreError: 394544:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8468.021137] Lustre: Skipped 1 previous similar message [ 8468.040317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.415631] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8472.456462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.561385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8473.976720] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8476.531969] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8478.373127] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8479.035395] Lustre: Permanently reactivating lustre-OST0000 [ 8479.040647] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8479.043092] Lustre: Skipped 2 previous similar messages [ 8506.335086] LustreError: 396422:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8508.669324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.787845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8508.791015] Lustre: Skipped 1 previous similar message [ 8510.747778] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8513.214794] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8526.309630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8526.310176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8526.311524] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8526.311528] Lustre: Skipped 3 previous similar messages [ 8526.316588] Lustre: Skipped 1 previous similar message [ 8526.320659] LustreError: Skipped 1 previous similar message [ 8526.999766] Lustre: server umount lustre-OST0000 complete [ 8528.864292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8528.868678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8530.387324] Lustre: server umount lustre-MDT0000 complete [ 8531.424571] LustreError: 394544:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.432465] LustreError: 394544:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8531.662524] LustreError: 396118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946260 with bad export cookie 16236584737325451061 [ 8531.663716] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8531.668361] LustreError: 396118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8531.677743] Lustre: 394536:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 8531.788979] Lustre: server umount lustre-MDT0001 complete [ 8535.418951] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8536.442494] Key type lgssc unregistered [ 8536.560367] LNet: 399262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8536.564611] LNetError: 399262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8536.577330] LNet: Removed LNI 192.168.203.114@tcp [ 8536.874129] Key type .llcrypt unregistered [ 8536.875280] Key type ._llcrypt unregistered [ 8542.723767] Key type ._llcrypt registered [ 8542.725481] Key type .llcrypt registered [ 8542.763636] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8548.772034] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8549.078543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8549.086673] alg: No test for adler32 (adler32-zlib) [ 8549.927835] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8550.014730] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8551.591156] Key type lgssc registered [ 8551.937980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.594353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8562.599799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.677400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8563.687634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8563.724317] Lustre: lustre-MDT0000: new disk, initializing [ 8563.747782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.754193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8565.007162] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8569.004022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.044760] Lustre: 402389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8569.064964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8569.067683] Lustre: Skipped 1 previous similar message [ 8569.099721] Lustre: lustre-MDT0001: new disk, initializing [ 8569.129208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8569.141800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8569.145883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8570.342524] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8573.939746] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.656197] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8577.910030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8578.007219] Lustre: lustre-OST0000: new disk, initializing [ 8578.009892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8578.012783] Lustre: 404360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8578.043486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.886782] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8583.575387] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8584.188268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8584.192798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8584.225554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8584.988070] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8585.061174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8586.438819] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8586.514668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8589.281905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8589.283482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8589.288379] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8589.290493] LustreError: Skipped 1 previous similar message [ 8589.294601] Lustre: Skipped 1 previous similar message [ 8593.295906] Lustre: server umount lustre-OST0000 complete [ 8594.912458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8594.916327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8594.922129] Lustre: Skipped 1 previous similar message [ 8594.925700] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8599.520775] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8599.522218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8600.663803] Lustre: server umount lustre-MDT0000 complete [ 8601.947395] LustreError: 402383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946331 with bad export cookie 5427814751666493338 [ 8601.950405] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.951272] LustreError: 402383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8602.068742] Lustre: server umount lustre-MDT0001 complete [ 8604.870258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8605.420491] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:25:33 (1784946333) [ 8607.704603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.875572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8609.204059] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8612.140475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.254523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8613.563603] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8615.847989] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8617.683716] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8619.844556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8621.814899] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8624.043277] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.642720] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8635.274299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8637.411064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8637.411169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8637.418526] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8637.421027] Lustre: Skipped 2 previous similar messages [ 8642.529391] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8642.532520] Lustre: Skipped 1 previous similar message [ 8642.767411] Lustre: server umount lustre-OST0000 complete [ 8647.649197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8647.655263] Lustre: Skipped 1 previous similar message [ 8652.769078] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8652.771015] Lustre: Skipped 3 previous similar messages [ 8658.399093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8658.449463] Lustre: server umount lustre-MDT0000 complete [ 8659.660252] LustreError: 406794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946388 with bad export cookie 5427814751666494220 [ 8659.663086] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.664165] LustreError: 406794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8659.772815] Lustre: server umount lustre-MDT0001 complete [ 8663.326771] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8664.310940] Key type lgssc unregistered [ 8664.442365] LNet: 410559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8664.446062] LNetError: 410559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8664.455643] LNet: Removed LNI 192.168.203.114@tcp [ 8664.787407] Key type .llcrypt unregistered [ 8664.788684] Key type ._llcrypt unregistered [ 8672.493163] Key type ._llcrypt registered [ 8672.494315] Key type .llcrypt registered [ 8672.527863] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8672.625577] Key type .llcrypt unregistered [ 8672.626934] Key type ._llcrypt unregistered [ 8679.468160] Key type ._llcrypt registered [ 8679.469585] Key type .llcrypt registered [ 8679.508387] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8679.827340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8679.833964] alg: No test for adler32 (adler32-zlib) [ 8680.682856] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8680.774309] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8682.367100] Key type lgssc registered [ 8682.707150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8685.125313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8685.131238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.270284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.571962] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8690.234936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8690.338603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8692.184273] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8694.459289] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8695.777902] LustreError: 412746:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.782898] LustreError: 412746:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 8696.501785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.615861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8697.640711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8697.799091] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8701.920731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8701.924473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8701.929699] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8702.946201] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8702.947393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8702.954911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8706.571987] Lustre: server umount lustre-OST0000 complete [ 8708.065029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8708.070238] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8708.073009] Lustre: Skipped 2 previous similar messages [ 8713.184647] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8713.187148] Lustre: Skipped 1 previous similar message [ 8713.938034] Lustre: server umount lustre-MDT0000 complete [ 8715.118328] LustreError: 412727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946444 with bad export cookie 2652935148032736117 [ 8715.121487] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8715.122775] LustreError: 412727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8715.238222] Lustre: server umount lustre-MDT0001 complete [ 8718.529350] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8719.443603] Key type lgssc unregistered [ 8719.562427] LNet: 415891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8719.565141] LNetError: 415891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8719.573276] LNet: Removed LNI 192.168.203.114@tcp [ 8719.873049] Key type .llcrypt unregistered [ 8719.874726] Key type ._llcrypt unregistered [ 8728.235761] Key type ._llcrypt registered [ 8728.236841] Key type .llcrypt registered [ 8728.274229] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8728.580254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8728.626899] alg: No test for adler32 (adler32-zlib) [ 8729.473443] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8729.557593] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8731.135151] Key type lgssc registered [ 8731.475184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8733.885899] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8733.893561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8749.343360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8754.463266] LustreError: 417257:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 8781.087148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.867267] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8785.088634] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8787.159022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8787.320812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.340368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8788.582885] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8791.268320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.381782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8792.629413] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8794.878313] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.597629] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8800.739518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8800.740199] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8800.744643] Lustre: Skipped 1 previous similar message [ 8804.871419] Lustre: server umount lustre-OST0000 complete [ 8806.880173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8806.882872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8806.886734] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8806.888827] Lustre: Skipped 1 previous similar message [ 8810.976506] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8810.977044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8812.180704] Lustre: server umount lustre-MDT0000 complete [ 8813.355156] LustreError: 417265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946542 with bad export cookie 11397894836166739729 [ 8813.358459] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.358622] LustreError: 417265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8813.462042] Lustre: server umount lustre-MDT0001 complete [ 8816.746122] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8817.709358] Key type lgssc unregistered [ 8817.840350] LNet: 420824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8817.843157] LNetError: 420824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8817.850269] LNet: Removed LNI 192.168.203.114@tcp [ 8818.120074] Key type .llcrypt unregistered [ 8818.121251] Key type ._llcrypt unregistered [ 8826.513763] Key type ._llcrypt registered [ 8826.514805] Key type .llcrypt registered [ 8826.549293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8826.854496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8826.896201] alg: No test for adler32 (adler32-zlib) [ 8827.747280] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8827.825893] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8829.399138] Key type lgssc registered [ 8829.724101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8832.125983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8832.132966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.278910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.599254] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8837.237108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.350757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8838.618644] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8840.916825] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8842.721589] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.987325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8845.110451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.980607] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8849.349820] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8858.405994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8863.200053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8863.203104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8863.208243] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8863.714241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8863.722084] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8863.725304] Lustre: Skipped 1 previous similar message [ 8865.095682] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 8866.903697] Lustre: server umount lustre-MDT0000 complete [ 8867.594141] LustreError: 422244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.14@tcp arrived at 1784946596 with bad export cookie 10241853780756019624 [ 8867.600154] LustreError: 422244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8868.169954] LustreError: 422197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946597 with bad export cookie 10241853780756019001 [ 8868.173864] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.174037] LustreError: 422197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8868.832481] LustreError: 422217:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8868.833020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8868.838322] LustreError: 422217:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 8868.843229] Lustre: Skipped 1 previous similar message [ 8868.848237] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8873.952232] LustreError: 422218:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.952378] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8873.959845] LustreError: 422218:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 8874.252811] Lustre: server umount lustre-OST0000 complete [ 8876.374860] Lustre: server umount lustre-MDT0001 complete [ 8879.713839] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8880.621410] Key type lgssc unregistered [ 8880.741309] LNet: 425918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.743599] LNetError: 425918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8880.754260] LNet: Removed LNI 192.168.203.114@tcp [ 8881.025513] Key type .llcrypt unregistered [ 8881.026633] Key type ._llcrypt unregistered [ 8889.554101] Key type ._llcrypt registered [ 8889.555345] Key type .llcrypt registered [ 8889.587970] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8889.958929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8889.977458] alg: No test for adler32 (adler32-zlib) [ 8890.816353] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8890.891850] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8892.463151] Key type lgssc registered [ 8892.782765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8895.057862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8895.063310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8896.195713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.349871] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8899.877764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8900.004373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8901.219564] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8903.353118] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.942560] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8906.968513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8907.067164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8908.947133] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8911.224163] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8921.447436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8925.664325] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8925.667569] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8925.672600] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 8926.691617] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 8926.693199] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8928.847409] Lustre: server umount lustre-MDT0001 complete [ 8931.808624] LustreError: 427309:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8931.809855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8931.809877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 8931.812751] LustreError: 427309:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8931.823195] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 8931.825611] Lustre: Skipped 2 previous similar messages [ 8936.412971] Lustre: server umount lustre-OST0000 complete [ 8936.930521] LustreError: 428802:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.916594] Lustre: server umount lustre-MDT0000 complete [ 8941.993277] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8942.983409] Key type lgssc unregistered [ 8943.116432] LNet: 431011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8943.119266] LNetError: 431011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8943.131341] LNet: Removed LNI 192.168.203.114@tcp [ 8943.433502] Key type .llcrypt unregistered [ 8943.434654] Key type ._llcrypt unregistered [ 8951.216341] Key type ._llcrypt registered [ 8951.217543] Key type .llcrypt registered [ 8951.251757] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8957.149109] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8957.579972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8957.593048] alg: No test for adler32 (adler32-zlib) [ 8958.448353] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8958.539128] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8960.127130] Key type lgssc registered [ 8960.522918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8973.195302] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8978.044648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8978.049665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8979.120197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8979.129111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8979.167550] Lustre: lustre-MDT0000: new disk, initializing [ 8979.187573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8979.193938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8980.342245] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8985.889354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8985.987062] Lustre: lustre-OST0000: new disk, initializing [ 8985.989796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8985.992916] Lustre: Skipped 1 previous similar message [ 8985.995568] Lustre: 436013:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.019396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.961121] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8988.039210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8988.043768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8988.055014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8991.786803] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8997.262245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.309640] Lustre: 434998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8997.349574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8997.389965] Lustre: lustre-MDT0001: new disk, initializing [ 8997.434335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8997.448526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8997.453788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8998.872369] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9002.579818] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9002.651341] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9004.031517] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9004.103668] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9006.056253] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 9006.059339] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9009.993728] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 9009.997290] Lustre: Skipped 1 previous similar message [ 9011.170450] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.114@tcp (stopping) [ 9011.174143] Lustre: Skipped 1 previous similar message [ 9011.860526] Lustre: server umount lustre-MDT0001 complete [ 9016.290173] LustreError: 435007:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9016.297141] LustreError: 435007:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 9021.409765] LustreError: 435003:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.417365] LustreError: 435003:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 9026.529028] LustreError: 435007:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.529226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9026.539768] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9030.416479] Lustre: server umount lustre-OST0000 complete [ 9031.650865] LustreError: 437337:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.942331] Lustre: server umount lustre-MDT0000 complete [ 9042.382477] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 9043.366276] Key type lgssc unregistered [ 9043.496381] LNet: 439758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9043.498876] LNetError: 439758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9043.507256] LNet: Removed LNI 192.168.203.114@tcp [ 9043.833208] Key type .llcrypt unregistered [ 9043.834418] Key type ._llcrypt unregistered [ 9056.515460] Key type ._llcrypt registered [ 9056.516431] Key type .llcrypt registered [ 9056.549965] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 9056.636151] Key type .llcrypt unregistered [ 9056.637087] Key type ._llcrypt unregistered [ 9062.706019] Key type ._llcrypt registered [ 9062.707065] Key type .llcrypt registered [ 9062.743354] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9063.107175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9063.129210] alg: No test for adler32 (adler32-zlib) [ 9063.973594] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 9064.063379] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 9065.639164] Key type lgssc registered [ 9065.994318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9074.191880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9074.197978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.278129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9075.287922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9075.328327] Lustre: lustre-MDT0000: new disk, initializing [ 9075.349236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.354707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9076.610863] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9080.543995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.582886] Lustre: 443106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9080.606770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9080.609233] Lustre: Skipped 1 previous similar message [ 9080.648942] Lustre: lustre-MDT0001: new disk, initializing [ 9080.679336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9080.689803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9080.693989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9081.940947] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9085.385799] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9086.980148] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9088.979592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9089.069173] Lustre: lustre-OST0000: new disk, initializing [ 9089.071368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9089.073499] Lustre: 445069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9089.097529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.903911] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9094.633256] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9097.206834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9097.211596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9097.246654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9102.306702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9102.308640] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9102.311964] Lustre: Skipped 2 previous similar messages [ 9102.317443] Lustre: Skipped 2 previous similar messages [ 9107.425933] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9107.428408] Lustre: Skipped 2 previous similar messages [ 9108.445624] Lustre: server umount lustre-MDT0000 complete [ 9109.665294] LustreError: 443098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946838 with bad export cookie 2502813916989736173 [ 9109.667251] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9109.669055] LustreError: 443098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9109.788345] Lustre: server umount lustre-MDT0001 complete [ 9121.377838] Lustre: server umount lustre-OST0000 complete [ 9124.968239] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:34:13 (1784946853) [ 9127.666831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.846856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9129.243540] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9132.123892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.268601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.789156] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9136.280585] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.118035] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.378283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.447877] Lustre: Found index 0 for lustre-OST0000, updating log [ 9153.513878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.385777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9157.702632] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9172.967770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9178.085408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.203.114@tcp failed: rc = -107 [ 9178.085986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9178.088728] LustreError: Skipped 1 previous similar message [ 9178.093322] Lustre: lustre-OST0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9178.095521] Lustre: Skipped 1 previous similar message [ 9180.629786] Lustre: server umount lustre-OST0000 complete [ 9193.202561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9193.203600] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9193.209127] Lustre: Skipped 2 previous similar messages [ 9193.211613] Lustre: Skipped 2 previous similar messages [ 9198.560952] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.114@tcp (stopping) [ 9198.563439] Lustre: Skipped 2 previous similar messages [ 9199.382894] Lustre: server umount lustre-MDT0000 complete [ 9200.692569] LustreError: 447078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946929 with bad export cookie 2502813916989737552 [ 9200.698090] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9200.822659] Lustre: server umount lustre-MDT0001 complete [ 9203.708286] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:35:31 (1784946931) [ 9213.599247] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [ 9783.481732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.661184] LustreError: 453516:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.114@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.671145] LustreError: 453516:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 9783.692415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9785.221995] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9788.123807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.242543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9789.591617] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9791.994811] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9793.750459] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9808.994837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9914.015311] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [ 9981.599339] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10042.015315] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10110.623244] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10172.063187] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10278.559408] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10338.975288] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10399.391623] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10528.415250] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10595.999234] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10671.775172] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10745.503542] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10805.919325] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10867.359239] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10928.799424] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10988.191394] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11112.095355] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11179.679378] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11241.119287] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11310.751125] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11372.191284] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11456.159409] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11517.599358] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11579.039561] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11583.135584] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.173.16@tcp: rc = -75 [11639.455339] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11643.551693] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.16@tcp: rc = -75 [11702.943662] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.173.17@tcp: rc = -75 [11761.311564] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11761.312183] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.17@tcp: rc = -75 [11821.727558] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11821.727918] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.173.18@tcp: rc = -75 [11880.095765] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.18@tcp: rc = -75 [11939.487655] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.173.19@tcp: rc = -75 [11966.111242] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [11999.903786] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.19@tcp: rc = -75 [12050.079509] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12111.519433] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12117.663784] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.20@tcp: rc = -75 [12117.670438] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) Skipped 1 previous similar message [12170.911511] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12291.743263] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12296.863838] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.173.22@tcp: rc = -75 [12296.873048] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) Skipped 2 previous similar messages [12471.967406] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12534.431446] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12592.799664] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) e11b1b54-fbce-48bd-b57d-8c3bf7fd2c15: can't add NID 111.222.142.24@tcp: rc = -75 [12592.806437] Lustre: 455330:0:(lustre_peer.c:124:class_add_uuid()) Skipped 4 previous similar messages [12595.871433] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12669.599338] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [12762.783451] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp