[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515816333 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003311] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006009] kvm-guest: setup PV IPIs [ 0.009577] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.012139] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.016016] *** VALIDATE selinux *** [ 0.024690] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029167] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032129] *** VALIDATE tmpfs *** [ 0.033481] *** VALIDATE proc *** [ 0.035266] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044265] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.046190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055298] rcu: Hierarchical SRCU implementation. [ 0.057462] smp: Bringing up secondary CPUs ... [ 0.058637] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062394] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144508] node 0 deferred pages initialised in 77ms [ 0.148110] devtmpfs: initialized [ 0.150307] x86/mm: Memory block size: 128MB [ 0.153835] gcov: version magic: 0x41383552 [ 0.156202] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161332] pinctrl core: initialized pinctrl subsystem [ 0.164249] [ 0.164804] ************************************************************* [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173013] ** ** [ 0.175012] ** This means that this kernel is built to expose internal ** [ 0.178013] ** IOMMU data structures, which may compromise security on ** [ 0.180011] ** your system. ** [ 0.182013] ** ** [ 0.184012] ** If you see this message and you are not debugging the ** [ 0.186013] ** kernel, report this immediately to your vendor! ** [ 0.190014] ** ** [ 0.192011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195015] ************************************************************* [ 0.198684] NET: Registered protocol family 16 [ 0.200516] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210028] cpuidle: using governor menu [ 0.211895] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214547] PCI: Using configuration type 1 for base access [ 0.216115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229155] cryptd: max_cpu_qlen set to 1000 [ 0.231215] ACPI: Added _OSI(Module Device) [ 0.233029] ACPI: Added _OSI(Processor Device) [ 0.235015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240831] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246540] ACPI: Interpreter enabled [ 0.248060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249011] ACPI: Using IOAPIC for interrupt routing [ 0.250108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265340] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270102] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278336] acpiphp: Slot [2] registered [ 0.280122] acpiphp: Slot [5] registered [ 0.281110] acpiphp: Slot [6] registered [ 0.283170] acpiphp: Slot [7] registered [ 0.284088] acpiphp: Slot [8] registered [ 0.286108] acpiphp: Slot [9] registered [ 0.287108] acpiphp: Slot [10] registered [ 0.288113] acpiphp: Slot [3] registered [ 0.290151] acpiphp: Slot [4] registered [ 0.291101] acpiphp: Slot [11] registered [ 0.292091] acpiphp: Slot [12] registered [ 0.294078] acpiphp: Slot [13] registered [ 0.295087] acpiphp: Slot [14] registered [ 0.296106] acpiphp: Slot [15] registered [ 0.298090] acpiphp: Slot [16] registered [ 0.299076] acpiphp: Slot [17] registered [ 0.300146] acpiphp: Slot [18] registered [ 0.301078] acpiphp: Slot [19] registered [ 0.303097] acpiphp: Slot [20] registered [ 0.304121] acpiphp: Slot [21] registered [ 0.306108] acpiphp: Slot [22] registered [ 0.307127] acpiphp: Slot [23] registered [ 0.309096] acpiphp: Slot [24] registered [ 0.310105] acpiphp: Slot [25] registered [ 0.312140] acpiphp: Slot [26] registered [ 0.313092] acpiphp: Slot [27] registered [ 0.315109] acpiphp: Slot [28] registered [ 0.316129] acpiphp: Slot [29] registered [ 0.319145] acpiphp: Slot [30] registered [ 0.321104] acpiphp: Slot [31] registered [ 0.322067] PCI host bridge to bus 0000:00 [ 0.324029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345550] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374455] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376850] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383781] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411780] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470563] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.516235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.524015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.563692] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.570018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.577016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.613012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.638573] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.646016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.675017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685465] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.688468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.691437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693534] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.695273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.699146] iommu: Default domain type: Passthrough [ 0.701499] SCSI subsystem initialized [ 0.703200] ACPI: bus type USB registered [ 0.705140] usbcore: registered new interface driver usbfs [ 0.707154] usbcore: registered new interface driver hub [ 0.710124] usbcore: registered new device driver usb [ 0.712236] pps_core: LinuxPPS API ver. 1 registered [ 0.714021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.717091] PTP clock support registered [ 0.719190] EDAC MC: Ver: 3.0.0 [ 0.722174] PCI: Using ACPI for IRQ routing [ 0.723862] NetLabel: Initializing [ 0.725011] NetLabel: domain hash size = 128 [ 0.727015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729081] NetLabel: unlabeled traffic allowed by default [ 0.731098] vgaarb: loaded [ 0.732373] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.734019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741236] clocksource: Switched to clocksource kvm-clock [ 0.856943] VFS: Disk quotas dquot_6.6.0 [ 0.858870] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.860924] *** VALIDATE ramfs *** [ 0.861918] *** VALIDATE hugetlbfs *** [ 0.863224] pnp: PnP ACPI init [ 0.865436] pnp: PnP ACPI: found 6 devices [ 0.887209] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.891415] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.893904] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.896417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899208] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.902327] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905409] NET: Registered protocol family 2 [ 0.908115] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913650] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917888] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.923531] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927346] TCP: Hash tables configured (established 65536 bind 65536) [ 0.930566] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.934542] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.937574] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942449] NET: Registered protocol family 1 [ 0.945518] RPC: Registered named UNIX socket transport module. [ 0.947855] RPC: Registered udp transport module. [ 0.949450] RPC: Registered tcp transport module. [ 0.950911] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.954456] NET: Registered protocol family 44 [ 0.956044] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.958495] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960252] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962430] PCI: CLS 0 bytes, default 64 [ 0.963935] Unpacking initramfs... [ 2.400493] debug: unmapping init [mem 0xffff9d13bcc54000-0xffff9d13bffbffff] [ 2.407511] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.410049] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414573] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.920473] Initialise system trusted keyrings [ 2.922194] Key type blacklist registered [ 2.925105] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936496] zbud: loaded [ 2.940087] *** VALIDATE nfs *** [ 2.941507] *** VALIDATE nfs4 *** [ 2.943472] pstore: using deflate compression [ 2.947740] Platform Keyring initialized [ 3.049466] NET: Registered protocol family 38 [ 3.051324] Key type asymmetric registered [ 3.053310] Asymmetric key parser 'x509' registered [ 3.055670] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.059144] io scheduler mq-deadline registered [ 3.060853] io scheduler kyber registered [ 3.062462] io scheduler bfq registered [ 3.064381] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.067985] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.071360] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.074156] ACPI: Power Button [PWRF] [ 3.079956] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.086836] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.099258] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.106529] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.124527] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.152482] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.180894] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.186411] Non-volatile memory driver v1.3 [ 3.188243] Linux agpgart interface v0.103 [ 3.223074] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.226177] vda: detected capacity change from 0 to 76681216 [ 3.240566] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.243036] vdb: detected capacity change from 0 to 1073741824 [ 3.257506] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.259990] vdc: detected capacity change from 0 to 2621440000 [ 3.273444] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275965] vdd: detected capacity change from 0 to 2621440000 [ 3.289377] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.291845] vde: detected capacity change from 0 to 4294967296 [ 3.305209] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308331] vdf: detected capacity change from 0 to 4294967296 [ 3.314167] libphy: Fixed MDIO Bus: probed [ 3.334734] usbcore: registered new interface driver usbserial_generic [ 3.338460] usbserial: USB Serial support registered for generic [ 3.340952] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.345702] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.347191] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.349556] mousedev: PS/2 mouse device common for all mice [ 3.352961] rtc_cmos 00:05: RTC can wake from S4 [ 3.356441] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.358576] rtc_cmos 00:05: registered as rtc0 [ 3.362498] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.363631] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.365961] intel_pstate: CPU model not supported [ 3.370133] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.374910] hid: raw HID events driver (C) Jiri Kosina [ 3.377158] usbcore: registered new interface driver usbhid [ 3.379502] usbhid: USB HID core driver [ 3.381072] drop_monitor: Initializing network drop monitor service [ 3.383773] Initializing XFRM netlink socket [ 3.386056] NET: Registered protocol family 10 [ 3.389179] Segment Routing with IPv6 [ 3.390689] NET: Registered protocol family 17 [ 3.393219] mpls_gso: MPLS GSO support [ 3.399113] RAS: Correctable Errors collector initialized. [ 3.401415] AVX version of gcm_enc/dec engaged. [ 3.403295] AES CTR mode by8 optimization enabled [ 3.488180] sched_clock: Marking stable (3488150668, 0)->(4429070596, -940919928) [ 3.492127] registered taskstats version 1 [ 3.494894] Loading compiled-in X.509 certificates [ 3.496860] zswap: loaded using pool lzo/zbud [ 3.521185] Key type big_key registered [ 3.533597] Key type encrypted registered [ 3.535384] ima: No TPM chip found, activating TPM-bypass! [ 3.537319] ima: Allocated hash algorithm: sha1 [ 3.539076] ima: No architecture policies found [ 3.540831] evm: Initialising EVM extended attributes: [ 3.542806] evm: security.selinux [ 3.544202] evm: security.ima [ 3.545194] evm: security.capability [ 3.546449] evm: HMAC attrs: 0x1 [ 3.548926] rtc_cmos 00:05: setting system clock to 2026-09-05 19:21:46 UTC (1788636106) [ 3.555392] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.558721] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.569164] Write protecting the kernel read-only data: 28672k [ 3.572624] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.575792] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.613845] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.622851] systemd[1]: Detected virtualization kvm. [ 3.625422] systemd[1]: Detected architecture x86-64. [ 3.627712] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.653901] systemd[1]: No hostname configured. [ 3.655178] systemd[1]: Set hostname to . [ 3.657434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.660129] systemd[1]: Initializing machine ID from random generator. [ 3.701598] random: ln: uninitialized urandom read (6 bytes read) [ 3.806350] random: systemd: uninitialized urandom read (16 bytes read) [ 3.808910] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.813142] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.817867] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.504072] device-mapper: uevent: version 1.0.3 [ 4.505843] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.294095] random: fast init done [ 5.295294] virtio_net virtio0 ens2: renamed from eth0 [ 5.319767] scsi host0: ata_piix [ 5.367338] scsi host1: ata_piix [ 5.369229] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.372250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.809938] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.051475] random: crng init done [ 10.053089] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.576772] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.786865] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.045554] SELinux: Disabled at runtime. [ 12.112687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.121779] systemd[1]: Detected virtualization kvm. [ 12.124014] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.655287] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.659059] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.663839] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.668373] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.672931] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.680441] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.688296] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ 12.799991] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.214980] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.637080] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.643979] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.813792] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.827700] EDAC sbridge: Ver: 1.1.2 [ 15.619046] Key type dns_resolver registered [ 15.929282] NFS: Registering the id_resolver key type [ 15.931247] Key type id_resolver registered [ 15.932787] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-server login: [ 98.714237] hrtimer: interrupt took 31227962 ns [ 101.564806] libcfs: loading out-of-tree module taints kernel. [ 101.653637] Key type ._llcrypt registered [ 101.657820] Key type .llcrypt registered [ 101.904030] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 129.645496] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 132.217789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 132.240807] alg: No test for adler32 (adler32-zlib) [ 134.051926] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 135.343160] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 137.303239] Key type lgssc registered [ 139.913418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 161.703336] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 208.380980] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 223.162812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 223.201250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.704960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.775229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.912599] Lustre: lustre-MDT0000: new disk, initializing [ 224.999320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.017678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.099108] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.139221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.285933] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.321678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.327636] Lustre: Skipped 1 previous similar message [ 245.480116] Lustre: lustre-MDT0001: new disk, initializing [ 245.657932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.702788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 245.711817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 250.490555] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.103623] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.084129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.312733] Lustre: lustre-OST0000: new disk, initializing [ 269.317393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.331840] Lustre: 8471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.511353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.745564] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.657994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 279.683557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 279.879779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 300.355676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 300.531619] Lustre: lustre-OST0001: new disk, initializing [ 300.534747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 300.547633] Lustre: 9547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 300.622929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 307.303562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 307.322940] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 307.443463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 308.455927] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 323.275729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 333.693489] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 340.609764] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing check_logdir /tmp/testlogs/ [ 345.741714] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing yml_node [ 350.474376] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 353.021635] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 355.716360] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 357.542032] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 15:27:38 EDT 2026 [ 375.226862] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 377.457919] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 394.726973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.728944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.744583] Lustre: Skipped 2 previous similar messages [ 394.760892] Lustre: Skipped 1 previous similar message [ 399.334549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 399.345603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.357566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.369296] Lustre: Skipped 1 previous similar message [ 400.736840] Lustre: server umount lustre-MDT0000 complete [ 404.656601] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636507 with bad export cookie 12700226282789139160 [ 404.656805] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.669162] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 404.962524] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.984367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 404.990649] Lustre: Skipped 3 previous similar messages [ 410.080958] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.081025] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.110124] LustreError: 6541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 410.126244] Lustre: Skipped 1 previous similar message [ 411.134076] Lustre: server umount lustre-MDT0001 complete [ 415.434352] Lustre: server umount lustre-OST0000 complete [ 420.073675] Lustre: server umount lustre-OST0001 complete [ 429.045748] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 442.460165] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 485.673982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.058694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 486.131366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 486.256298] Lustre: lustre-MDT0000: new disk, initializing [ 486.356266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.373712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 491.784681] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 504.763683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.995958] Lustre: 15102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 505.100912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 505.110766] Lustre: Skipped 1 previous similar message [ 505.238685] Lustre: lustre-MDT0001: new disk, initializing [ 505.456753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.529624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 505.551101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 510.939326] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 523.440166] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.921227] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 538.811602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.220863] Lustre: lustre-OST0000: new disk, initializing [ 539.230436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 539.239657] Lustre: 17074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 539.354559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.055912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 541.081605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 541.223841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 547.780845] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 559.537523] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.686585] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 567.021800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 572.864857] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 573.267811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 576.994559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 577.007378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.015144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.021644] Lustre: Skipped 2 previous similar messages [ 577.060520] Lustre: Skipped 1 previous similar message [ 581.754795] Lustre: server umount lustre-OST0000 complete [ 587.236401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.240995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 587.283284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 587.292568] LustreError: Skipped 1 previous similar message [ 587.320719] Lustre: Skipped 1 previous similar message [ 591.931244] Lustre: server umount lustre-MDT0000 complete [ 592.354774] LustreError: 15110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.419583] LustreError: 15110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 596.170079] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636699 with bad export cookie 12700226282789140693 [ 596.178680] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.187868] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 596.456276] Lustre: server umount lustre-MDT0001 complete [ 601.437632] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:31:41 (1788636701) [ 604.137675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 606.662839] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:31:46 (1788636706) [ 613.922949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.631942] LustreError: 19485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.705622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.004562] LustreError: 19486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.051266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 624.032749] LustreError: 19485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.117761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.400298] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 645.706809] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 653.866926] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 662.134275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.859662] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 680.484901] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 696.808233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 696.815980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.827518] Lustre: Skipped 1 previous similar message [ 696.840335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.713376] Lustre: server umount lustre-OST0000 complete [ 718.305442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 718.720884] Lustre: server umount lustre-MDT0000 complete [ 720.352536] LustreError: 19481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.370915] LustreError: 19481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 723.905423] LustreError: 19466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636826 with bad export cookie 12700226282789141582 [ 723.911128] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.915886] LustreError: 19466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 724.289863] Lustre: server umount lustre-MDT0001 complete [ 737.694248] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 740.750995] Key type lgssc unregistered [ 741.224960] LNet: 23041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.234676] LNetError: 23041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.266252] LNet: Removed LNI 192.168.204.109@tcp [ 742.369167] Key type .llcrypt unregistered [ 742.374511] Key type ._llcrypt unregistered [ 769.348822] Key type ._llcrypt registered [ 769.351581] Key type .llcrypt registered [ 769.443313] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 770.738607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.855939] alg: No test for adler32 (adler32-zlib) [ 771.999685] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 772.369603] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 774.071838] Key type lgssc registered [ 775.034620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 783.644910] loop: module loaded [ 783.747850] loop0: detected capacity change from 0 to 204800000 [ 784.011024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.022346] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.409750] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 791.003607] loop0: detected capacity change from 0 to 204800000 [ 791.208111] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 791.215095] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 791.227306] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.193219] loop0: detected capacity change from 0 to 204800000 [ 804.274448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.318618] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.678444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 805.826961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 805.936191] Lustre: lustre-MDT0000: new disk, initializing [ 806.090515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.134988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 811.318493] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 821.766815] loop1: detected capacity change from 0 to 204800000 [ 829.385055] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.678448] Lustre: lustre-OST07c6: new disk, initializing [ 829.687645] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 829.700828] Lustre: Skipped 1 previous similar message [ 829.708909] Lustre: 26241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 829.834209] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 836.738425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 836.748418] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 836.810779] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 838.381262] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 844.178986] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 844.198828] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 844.212814] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 844.224398] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 844.235500] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 844.256763] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 861.699622] Lustre: 25228:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 861.709465] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 861.718420] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 861.731282] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 861.747573] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 861.766588] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 867.348195] Lustre: 25228:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 867.362463] Lustre: 25228:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 867.369963] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 867.388076] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.400045] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 867.412579] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.422220] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 867.440610] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.454057] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 867.463922] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.472474] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 867.485508] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.857130] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 870.869582] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 870.876899] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 870.885699] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.893126] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 870.910197] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.929826] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 870.944884] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.959486] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 870.967980] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.977359] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 870.983702] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.867633] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 879.879897] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 879.885101] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 879.898784] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 879.909354] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 879.918978] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 879.933142] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 879.946063] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 879.962546] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 879.970441] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 879.978840] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 879.987593] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 889.922044] Lustre: 25227:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 889.944203] Lustre: 25227:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 889.956674] Lustre: 25227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 889.972803] Lustre: 25227:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 889.984179] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 890.004925] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 890.017194] Lustre: 25227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 890.026669] Lustre: 25227:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 890.034235] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 890.039066] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 890.048286] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 890.054702] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 944.105194] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.115471] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.128791] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 946.719934] Lustre: server umount lustre-OST07c6 complete [ 957.679196] Lustre: server umount lustre-MDT0000 complete [ 976.740523] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 979.573634] Key type lgssc unregistered [ 979.904724] LNet: 28802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.912516] LNetError: 28802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.934203] LNet: Removed LNI 192.168.204.109@tcp [ 981.064866] Key type .llcrypt unregistered [ 981.072695] Key type ._llcrypt unregistered [ 1004.307919] Key type ._llcrypt registered [ 1004.309614] Key type .llcrypt registered [ 1004.428827] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1005.444123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.527659] alg: No test for adler32 (adler32-zlib) [ 1006.754532] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1006.996380] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1008.728347] Key type lgssc registered [ 1009.718849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1019.240647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1019.272376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.937451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.054944] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1030.120370] LustreError: 30069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.161368] LustreError: 30069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1035.240512] LustreError: 30070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.171203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.501306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.959713] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1050.754524] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1058.247547] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1066.278225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1066.701057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.933417] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1082.962681] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1090.235605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1116.131325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.140088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.159181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.172458] Lustre: Skipped 1 previous similar message [ 1120.511687] Lustre: server umount lustre-OST0000 complete [ 1126.372218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.389612] Lustre: Skipped 1 previous similar message [ 1126.398310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.402855] Lustre: Skipped 1 previous similar message [ 1131.234357] Lustre: server umount lustre-MDT0000 complete [ 1131.488767] LustreError: 32576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.520987] LustreError: 32576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1135.368892] LustreError: 30051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637238 with bad export cookie 12454003276666182037 [ 1135.370211] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.376975] LustreError: 30051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1135.801588] Lustre: server umount lustre-MDT0001 complete [ 1148.764800] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1152.221653] Key type lgssc unregistered [ 1152.556035] LNet: 33831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.563755] LNetError: 33831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.591499] LNet: Removed LNI 192.168.204.109@tcp [ 1153.557000] Key type .llcrypt unregistered [ 1153.559898] Key type ._llcrypt unregistered [ 1178.615868] Key type ._llcrypt registered [ 1178.617618] Key type .llcrypt registered [ 1178.790467] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1180.225480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.381540] alg: No test for adler32 (adler32-zlib) [ 1181.948493] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1182.341867] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1184.023202] Key type lgssc registered [ 1185.413775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.489495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.505818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.333692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.187528] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1206.755319] LustreError: 35100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.770729] LustreError: 35100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1211.271704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.683366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.049354] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1224.220471] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1229.754366] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1237.540151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.819694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.928607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1245.017436] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1254.916963] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1271.791114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.811625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.811746] Lustre: Skipped 1 previous similar message [ 1274.386187] Lustre: server umount lustre-OST0000 complete [ 1282.021064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.035729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.054659] Lustre: Skipped 1 previous similar message [ 1282.076169] Lustre: Skipped 2 previous similar messages [ 1285.978986] Lustre: server umount lustre-MDT0000 complete [ 1287.140252] LustreError: 35100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.173548] LustreError: 35100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1290.378117] LustreError: 35079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637393 with bad export cookie 9112043642955102451 [ 1290.384333] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.394830] LustreError: 35079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1290.831700] Lustre: server umount lustre-MDT0001 complete [ 1304.047566] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1308.034274] Key type lgssc unregistered [ 1308.463890] LNet: 38658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.475580] LNetError: 38658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.508701] LNet: Removed LNI 192.168.204.109@tcp [ 1309.578944] Key type .llcrypt unregistered [ 1309.585039] Key type ._llcrypt unregistered [ 1335.794299] Key type ._llcrypt registered [ 1335.809688] Key type .llcrypt registered [ 1335.954300] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1337.285960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.342676] alg: No test for adler32 (adler32-zlib) [ 1338.446843] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1338.706822] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1340.399172] Key type lgssc registered [ 1341.890984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.287656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.332556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.812447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.224937] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1362.404737] LustreError: 40072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.421706] LustreError: 40072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1367.524206] LustreError: 40071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.737968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.212579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.496278] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1383.145127] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.421360] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1397.442274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.937554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.676537] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1412.357760] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1422.528936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1429.831414] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 1430.502395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1430.514375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.532648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.059290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.060960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.069433] Lustre: Skipped 1 previous similar message [ 1433.082948] Lustre: Skipped 2 previous similar messages [ 1438.177028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.189363] Lustre: Skipped 3 previous similar messages [ 1443.312568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.322605] Lustre: Skipped 1 previous similar message [ 1444.319220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1444.626766] Lustre: server umount lustre-MDT0000 complete [ 1445.149748] LustreError: 40067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.424104] LustreError: 40072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.442951] LustreError: 40072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1449.087597] LustreError: 40053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637552 with bad export cookie 18442053599271076675 [ 1449.088877] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.098850] LustreError: 40053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1453.275191] LustreError: 40066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.277448] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [ 1453.314821] LustreError: 40066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1453.330552] Lustre: Skipped 1 previous similar message [ 1453.537733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.540344] Lustre: server umount lustre-MDT0001 complete [ 1470.597070] Lustre: server umount lustre-OST0000 complete [ 1490.879643] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1493.861958] Key type lgssc unregistered [ 1494.123801] LNet: 43952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.130776] LNetError: 43952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.157122] LNet: Removed LNI 192.168.204.109@tcp [ 1495.236919] Key type .llcrypt unregistered [ 1495.239161] Key type ._llcrypt unregistered [ 1524.964923] Key type ._llcrypt registered [ 1524.970683] Key type .llcrypt registered [ 1525.200453] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1526.375810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.594979] alg: No test for adler32 (adler32-zlib) [ 1527.723213] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1527.917625] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1529.577627] Key type lgssc registered [ 1530.844862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1540.345132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1540.388455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.046408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.382397] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1551.333116] LustreError: 45370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1551.360975] LustreError: 45370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1556.449798] LustreError: 45369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.912458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.688312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1563.645636] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1572.611354] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1579.019951] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1586.138159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.397757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.439092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1591.721907] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1599.820462] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.589649] Lustre: lustre-OST0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 1613.287515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1613.308626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.324777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.306964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1614.323502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.350988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.359156] Lustre: Skipped 1 previous similar message [ 1615.906674] Lustre: server umount lustre-OST0000 complete [ 1635.296745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1635.314071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.336143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.907190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.913149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.945819] Lustre: Skipped 1 previous similar message [ 1641.577312] Lustre: server umount lustre-MDT0000 complete [ 1645.026373] LustreError: 45366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.058606] LustreError: 45366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1645.971410] LustreError: 45350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637748 with bad export cookie 11983121922641777417 [ 1645.975770] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.022604] LustreError: 45350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1660.384044] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1660.733434] Lustre: server umount lustre-MDT0001 complete [ 1673.170861] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1675.902825] Key type lgssc unregistered [ 1676.182786] LNet: 49079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.191560] LNetError: 49079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.203702] LNet: Removed LNI 192.168.204.109@tcp [ 1677.156958] Key type .llcrypt unregistered [ 1677.158994] Key type ._llcrypt unregistered [ 1703.612417] Key type ._llcrypt registered [ 1703.614693] Key type .llcrypt registered [ 1703.712895] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1704.831935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1704.911146] alg: No test for adler32 (adler32-zlib) [ 1706.004167] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1706.262720] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1707.927329] Key type lgssc registered [ 1709.614380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.374572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.440579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.176582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.837428] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1730.532516] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.557440] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1731.553324] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.075582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.568203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.015699] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1747.216653] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.478476] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1761.171642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.479392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.602436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1767.836405] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1775.345944] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.222499] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1788.786808] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1801.140468] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0515h51m40s [ 1803.031127] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1805.793187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.797570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.829033] Lustre: Skipped 1 previous similar message [ 1806.895197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.909333] Lustre: Skipped 4 previous similar messages [ 1808.692178] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 1811.440539] Lustre: server umount lustre-MDT0000 complete [ 1812.970831] LustreError: 50494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.988614] LustreError: 50494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1815.413975] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1815.822408] LustreError: 51505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.981934] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.007531] LustreError: 50493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1822.546598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.681522] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1822.995783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.069925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1824.046795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1828.696994] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1832.481336] Lustre: DEBUG MARKER: Wait for df (36714) ... [ 1834.360855] Lustre: DEBUG MARKER: done [ 1839.600210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.604977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.618118] Lustre: Skipped 2 previous similar messages [ 1839.631535] Lustre: Skipped 2 previous similar messages [ 1844.705368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.715889] Lustre: Skipped 1 previous similar message [ 1845.366752] Lustre: server umount lustre-OST0000 complete [ 1849.825472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.828361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.854079] Lustre: Skipped 1 previous similar message [ 1855.977560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.991640] Lustre: Skipped 4 previous similar messages [ 1864.159103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1864.425496] Lustre: server umount lustre-MDT0000 complete [ 1868.269001] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.301436] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1869.516563] LustreError: 53125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637972 with bad export cookie 4203165128830201640 [ 1869.528607] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.535534] LustreError: 53125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1870.050988] Lustre: server umount lustre-MDT0001 complete [ 1882.031460] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1885.160258] Key type lgssc unregistered [ 1885.575959] LNet: 55422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.582446] LNetError: 55422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.605188] LNet: Removed LNI 192.168.204.109@tcp [ 1886.732431] Key type .llcrypt unregistered [ 1886.734627] Key type ._llcrypt unregistered [ 1934.565577] Key type ._llcrypt registered [ 1934.570519] Key type .llcrypt registered [ 1934.723959] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1935.760897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.908752] alg: No test for adler32 (adler32-zlib) [ 1937.137389] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 1937.469682] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1939.135190] Key type lgssc registered [ 1940.302080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.741488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1950.765283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.103647] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1952.131933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1952.321832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.375483] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1971.503230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.629521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1971.686938] Lustre: 57627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1971.960979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.091313] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1989.102213] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1996.810574] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2005.330152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2005.534653] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2005.739854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.278119] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2014.882635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2021.768740] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2030.195562] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0515h55m30s [ 2031.720921] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2040.605554] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2044.571550] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2044.605911] LustreError: 57633:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2064.722686] Lustre: lustre-MDT0000: Client 83f8ffbd-58df-455c-905d-8f67240f0a53 (at 192.168.204.9@tcp) reconnecting [ 2084.687991] LustreError: 57633:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2087.732655] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2091.556008] Lustre: DEBUG MARKER: done [ 2101.728855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2101.746109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.777089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.247310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2102.258784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.538581] Lustre: server umount lustre-OST0000 complete [ 2110.432478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2110.442074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.465512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.474645] Lustre: Skipped 2 previous similar messages [ 2111.592156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.607253] Lustre: Skipped 2 previous similar messages [ 2113.633900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.653187] Lustre: Skipped 1 previous similar message [ 2120.165235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.182667] Lustre: Skipped 1 previous similar message [ 2124.259044] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2124.459438] Lustre: server umount lustre-MDT0000 complete [ 2128.586802] LustreError: 57620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638231 with bad export cookie 8210375864366891976 [ 2128.594156] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.597348] LustreError: 57620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.993291] Lustre: server umount lustre-MDT0001 complete [ 2140.556565] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2143.993788] Key type lgssc unregistered [ 2144.310297] LNet: 61966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.317514] LNetError: 61966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.339289] LNet: Removed LNI 192.168.204.109@tcp [ 2145.450805] Key type .llcrypt unregistered [ 2145.453395] Key type ._llcrypt unregistered [ 2196.349541] Key type ._llcrypt registered [ 2196.355066] Key type .llcrypt registered [ 2196.466746] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2198.337680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.430128] alg: No test for adler32 (adler32-zlib) [ 2199.568779] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2200.023814] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2201.711208] Key type lgssc registered [ 2203.144912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.734576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.756493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.083904] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2208.102468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2208.311101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.583423] Lustre: Failing over lustre-MDT0000 [ 2209.896441] Lustre: server umount lustre-MDT0000 complete [ 2218.148305] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:58:38 (1788638318) [ 2226.227935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.779824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.779195] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2240.806849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.887472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2240.915200] Lustre: 64760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2241.165248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.272751] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2254.098521] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2259.643762] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2266.668056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2266.891392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2267.096516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.927307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2274.242364] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2282.089948] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2288.681763] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2297.317933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2297.325801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.338509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.850428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.851355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.860051] Lustre: Skipped 1 previous similar message [ 2298.874194] Lustre: Skipped 1 previous similar message [ 2301.774329] Lustre: server umount lustre-MDT0000 complete [ 2306.023603] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638408 with bad export cookie 836058545709889043 [ 2306.037142] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.047539] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2306.568124] Lustre: server umount lustre-MDT0001 complete [ 2310.476724] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2315.510551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2325.473872] Lustre: 63688:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638412/real 1788638412] req@00000000147836e7 x1875523196199680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2325.526398] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.553136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.263264] Lustre: 63689:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638414/real 1788638414] req@000000009e236110 x1875523196199808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2330.591306] Lustre: 63687:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638417/real 1788638417] req@000000009016d998 x1875523196199936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2332.344941] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.375256] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2332.562274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.770720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2333.472162] Lustre: 63687:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638419/real 1788638419] req@00000000991502d4 x1875523196200192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2334.562145] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.585537] LustreError: 68534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.609568] LustreError: 68534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2337.631782] Lustre: 63688:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638424/real 1788638424] req@000000009b50ef98 x1875523196200320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2338.289563] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2340.849570] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.893267] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2345.959214] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.974595] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2348.612611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.005742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2353.701549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2354.157634] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2362.389252] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2368.163143] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2374.632348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.637725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.648500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.650790] Lustre: Skipped 2 previous similar messages [ 2374.665965] Lustre: Skipped 1 previous similar message [ 2375.008720] Lustre: server umount lustre-MDT0000 complete [ 2378.925989] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638481 with bad export cookie 836058545709891318 [ 2378.928196] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.935753] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2379.316545] Lustre: server umount lustre-MDT0001 complete [ 2386.785906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2396.065264] Lustre: 63687:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788638482/real 1788638482] req@00000000fbcc6ebe x1875523196226688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788638498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2396.098138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.958852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.641365] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.729579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.789129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2408.578599] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2415.079406] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2415.089751] Lustre: Skipped 1 previous similar message [ 2417.907623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.386261] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2423.799505] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2431.869971] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.056802] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2448.865916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2448.882426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.909828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.918384] Lustre: Skipped 1 previous similar message [ 2451.252727] Lustre: server umount lustre-MDT0000 complete [ 2454.503158] LustreError: 71229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.521045] LustreError: 71229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2455.189362] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638558 with bad export cookie 836058545709892697 [ 2455.191880] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.208267] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.459309] Lustre: server umount lustre-MDT0001 complete [ 2462.957361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2467.395251] Lustre: DEBUG MARKER: files compared the same [ 2479.378648] Lustre: server umount lustre-OST0000 complete [ 2497.166389] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2500.000380] Key type lgssc unregistered [ 2500.437749] LNet: 74503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.456417] LNetError: 74503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2500.478311] LNet: Removed LNI 192.168.204.109@tcp [ 2501.396720] Key type .llcrypt unregistered [ 2501.403283] Key type ._llcrypt unregistered [ 2529.243459] Key type ._llcrypt registered [ 2529.254311] Key type .llcrypt registered [ 2529.396596] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2530.720272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.751158] alg: No test for adler32 (adler32-zlib) [ 2531.886982] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2532.143370] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2533.872989] Key type lgssc registered [ 2535.081591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2543.875918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.889594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.315331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.660274] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug +malloc all [ 2555.872533] LustreError: 75920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.894277] LustreError: 75920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2558.831518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.144524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2564.143515] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug +malloc all [ 2572.639036] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2579.020986] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2586.257516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2586.861148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.968073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2594.025260] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug +malloc all [ 2603.763660] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2614.752232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.758889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.761875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2614.780394] Lustre: Skipped 1 previous similar message [ 2619.177935] Lustre: server umount lustre-OST0000 complete [ 2625.019215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.023972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.039934] Lustre: Skipped 1 previous similar message [ 2625.061056] Lustre: Skipped 2 previous similar messages [ 2630.113840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.133025] Lustre: Skipped 1 previous similar message [ 2634.210955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.221508] Lustre: Skipped 1 previous similar message [ 2638.303175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2638.521977] Lustre: server umount lustre-MDT0000 complete [ 2639.341594] LustreError: 75914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.363141] LustreError: 75914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2642.986750] LustreError: 75899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638745 with bad export cookie 11178184240422073148 [ 2642.994885] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.369333] Lustre: server umount lustre-MDT0001 complete [ 2655.563602] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2658.612764] Key type lgssc unregistered [ 2659.065870] LNet: 79453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.079847] LNetError: 79453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.101654] LNet: Removed LNI 192.168.204.109@tcp [ 2659.936716] Key type .llcrypt unregistered [ 2659.938883] Key type ._llcrypt unregistered [ 2682.825967] Key type ._llcrypt registered [ 2682.827993] Key type .llcrypt registered [ 2682.943274] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2683.862139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.910963] alg: No test for adler32 (adler32-zlib) [ 2685.026471] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2685.272071] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2687.015508] Key type lgssc registered [ 2688.025733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.990807] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2696.020593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.585329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2716.703609] LustreError: 80842:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.109@tcp: failed processing log, type 4: rc = -110 [ 2743.455266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.477320] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2759.573357] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2767.649079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.787968] Lustre: *** cfs_fail_loc=706, val=0*** [ 2768.142713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.218979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2772.779265] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2773.479802] LustreError: 80868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.501293] LustreError: 80868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2778.611886] LustreError: 81928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.644624] LustreError: 81928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2782.100656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.535329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.241482] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2795.964131] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.500527] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2808.295079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2808.310059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.358251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.802316] Lustre: server umount lustre-OST0000 complete [ 2818.529041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.529816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.550584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.568971] Lustre: Skipped 2 previous similar messages [ 2818.586327] Lustre: Skipped 2 previous similar messages [ 2822.878230] Lustre: server umount lustre-MDT0000 complete [ 2823.648313] LustreError: 81932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.670813] LustreError: 81932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2827.079917] LustreError: 80848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638930 with bad export cookie 18096514249642898455 [ 2827.096001] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.098269] LustreError: 80848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2827.459672] Lustre: server umount lustre-MDT0001 complete [ 2837.450441] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2839.934250] Key type lgssc unregistered [ 2840.188552] LNet: 84456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.194554] LNetError: 84456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.241772] LNet: Removed LNI 192.168.204.109@tcp [ 2841.132670] Key type .llcrypt unregistered [ 2841.136291] Key type ._llcrypt unregistered [ 2866.263619] Key type ._llcrypt registered [ 2866.266410] Key type .llcrypt registered [ 2866.342908] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2867.714885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2867.745709] alg: No test for adler32 (adler32-zlib) [ 2868.837195] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2869.057080] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2870.735217] Key type lgssc registered [ 2871.833138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.878662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.891101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.773312] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2893.815893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.422251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.006934] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2899.428444] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.439237] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2904.544645] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.567862] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2907.644401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.918888] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.940448] LustreError: 86555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2907.951296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.249037] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.264811] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2914.058810] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2918.381422] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.413627] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2922.826243] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.839861] Lustre: MGS: Client 6b0e5c80-b0d1-440e-878f-9de2e319147e (at 0@lo) reconnecting [ 2922.855189] LustreError: 86571:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2923.055262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.162057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2927.476609] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2943.460968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.464712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.468420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.484495] Lustre: Skipped 1 previous similar message [ 2946.606338] Lustre: server umount lustre-OST0000 complete [ 2953.696876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.719495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.725114] Lustre: Skipped 1 previous similar message [ 2956.878288] Lustre: server umount lustre-MDT0000 complete [ 2958.834832] LustreError: 88101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.871400] LustreError: 88101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2961.259807] Lustre: server umount lustre-MDT0001 complete [ 2965.344425] Lustre: server umount MGS complete [ 2975.397104] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:11:16 (1788639076) [ 3005.170549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3015.191680] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3054.647090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.036981] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3069.528995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.765781] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3069.859334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3069.991973] Lustre: lustre-MDT0001: new disk, initializing [ 3070.089547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.115388] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3070.129458] Lustre: 93726:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3070.143474] LustreError: 93726:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3074.637374] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3085.961330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3086.309635] Lustre: lustre-OST0000: new disk, initializing [ 3086.315544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3086.324310] Lustre: 94491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3086.413856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.143041] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3094.562315] Lustre: 94865:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009ad56aac x1875523898745728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788639213 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3102.795809] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3110.027796] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3110.037334] Lustre: MGS: Client c9999322-2395-46d1-8f78-43863b64e231 (at 0@lo) reconnecting [ 3110.102380] Lustre: 92992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3110.114730] LustreError: 93712:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3110.129126] LustreError: 93712:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3110.158498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3110.246872] Lustre: lustre-MDT0000: new disk, initializing [ 3110.389721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.422288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3110.441522] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3111.521642] Lustre: 95560:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000009a680c8 x1875523898752512/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788639230 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3115.184776] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3116.642371] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3120.635325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3120.647446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3120.739919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3146.218108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.220861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.233243] Lustre: Skipped 2 previous similar messages [ 3146.244285] Lustre: Skipped 1 previous similar message [ 3150.943669] Lustre: server umount lustre-OST0000 complete [ 3156.450125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3156.450287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.456949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.490952] Lustre: Skipped 1 previous similar message [ 3161.576083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.582833] Lustre: Skipped 2 previous similar messages [ 3169.247099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3169.448825] Lustre: server umount lustre-MDT0000 complete [ 3171.812364] LustreError: 93701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.843451] LustreError: 93701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3180.397819] Lustre: server umount lustre-MDT0001 complete [ 3184.550480] Lustre: server umount MGS complete [ 3196.658306] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:14:56 (1788639296) [ 3214.728897] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3217.987600] Key type lgssc unregistered [ 3218.392402] LNet: 98058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.405929] LNetError: 98058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.436344] LNet: Removed LNI 192.168.204.109@tcp [ 3219.471433] Key type .llcrypt unregistered [ 3219.473532] Key type ._llcrypt unregistered [ 3238.381816] Key type ._llcrypt registered [ 3238.384743] Key type .llcrypt registered [ 3238.538694] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3239.550956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3239.862126] alg: No test for adler32 (adler32-zlib) [ 3241.077883] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3241.329274] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3243.008024] Key type lgssc registered [ 3244.228827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.404893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.449842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.859390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.759137] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3263.467302] LustreError: 99275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.487049] LustreError: 99275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3267.085267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.600476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3272.327442] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3281.081802] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3287.910148] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3296.591502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.064836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.109648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3303.149246] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3310.018270] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.848309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3322.860979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.870940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3322.872142] Lustre: Skipped 1 previous similar message [ 3322.877533] Lustre: Skipped 1 previous similar message [ 3327.436845] Lustre: server umount lustre-OST0000 complete [ 3333.092647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.109802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.209674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.225282] Lustre: Skipped 2 previous similar messages [ 3342.307733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.399158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3346.658244] Lustre: server umount lustre-MDT0000 complete [ 3347.430455] LustreError: 99275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.470391] LustreError: 99275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3350.549466] LustreError: 99257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639453 with bad export cookie 13072402930537595309 [ 3350.567526] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.576773] LustreError: 99257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3350.852791] Lustre: server umount lustre-MDT0001 complete [ 3361.298167] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3364.980426] Key type lgssc unregistered [ 3365.501986] LNet: 102811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3365.513766] LNetError: 102811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3365.535535] LNet: Removed LNI 192.168.204.109@tcp [ 3367.236081] Key type .llcrypt unregistered [ 3367.242773] Key type ._llcrypt unregistered [ 3386.338533] Key type ._llcrypt registered [ 3386.342503] Key type .llcrypt registered [ 3386.483580] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3387.988243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3388.027765] alg: No test for adler32 (adler32-zlib) [ 3389.100527] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3389.413611] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3391.079521] Key type lgssc registered [ 3392.306756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.684091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3399.696415] LustreError: 103894:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3399.700454] LustreError: 103895:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3399.722952] LustreError: 103894:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3399.730985] LustreError: 103894:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3399.747379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.749100] LustreError: 103894:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3401.226449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.569765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.864517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3414.169962] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3419.508433] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing lsmod [ 3422.094245] LustreError: 105101:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3422.095697] LustreError: 105100:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3422.114503] LustreError: 105101:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3422.123530] LustreError: 105101:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3422.131921] LustreError: 105101:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3422.136465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.483089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.229162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.236400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.247369] Lustre: Skipped 2 previous similar messages [ 3425.256199] Lustre: Skipped 1 previous similar message [ 3430.368772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.377164] Lustre: Skipped 3 previous similar messages [ 3431.640892] Lustre: server umount lustre-MDT0000 complete [ 3434.472742] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.503798] LustreError: 103932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3435.772621] LustreError: 103914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639538 with bad export cookie 1894988005838153039 [ 3435.784871] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.785616] LustreError: 103914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3436.639340] LustreError: 104363:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003ee358e5 x1875524443983232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3436.926114] Lustre: server umount lustre-MDT0001 complete [ 3451.802083] Lustre: server umount lustre-OST0000 complete [ 3461.920399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.333433] LustreError: 106166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.409402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.311832] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3467.747493] LustreError: 106167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.871218] LustreError: 106166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.986436] LustreError: 106167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.262400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.878175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.534784] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3493.379159] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3499.562977] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3505.901524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.306292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.316929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3512.410339] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3519.849483] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3533.283490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3533.292132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.313068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.308400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3534.316982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.172634] Lustre: server umount lustre-OST0000 complete [ 3539.430051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.433054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.445957] Lustre: Skipped 1 previous similar message [ 3539.457940] Lustre: Skipped 3 previous similar messages [ 3544.553799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.561163] Lustre: Skipped 1 previous similar message [ 3545.655390] Lustre: server umount lustre-MDT0000 complete [ 3549.460688] LustreError: 106148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639652 with bad export cookie 1894988005838153739 [ 3549.471728] LustreError: 106148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3549.484407] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.853170] Lustre: server umount lustre-MDT0001 complete [ 3560.196504] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3563.913945] Key type lgssc unregistered [ 3564.312823] LNet: 109718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3564.326588] LNetError: 109718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3564.352515] LNet: Removed LNI 192.168.204.109@tcp [ 3565.298480] Key type .llcrypt unregistered [ 3565.300671] Key type ._llcrypt unregistered [ 3590.076278] Key type ._llcrypt registered [ 3590.079340] Key type .llcrypt registered [ 3590.201833] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3591.311026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3591.422627] alg: No test for adler32 (adler32-zlib) [ 3592.453492] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3592.673642] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3594.335228] Key type lgssc registered [ 3595.419951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3603.644191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3603.688194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.191518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.228363] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3615.715873] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.745424] LustreError: 111132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3620.832340] LustreError: 111131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.798827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.315514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3628.971711] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3636.952850] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3643.820386] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3651.511745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.872209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.696965] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3668.023532] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3675.450909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3683.922869] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3687.201087] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3694.050688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.057546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.073794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.098873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.115104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.120244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.152748] Lustre: Skipped 1 previous similar message [ 3697.486366] Lustre: server umount lustre-OST0000 complete [ 3705.312658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3705.319512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.339517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.460031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.475940] Lustre: Skipped 3 previous similar messages [ 3716.063139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3716.270683] Lustre: server umount lustre-MDT0000 complete [ 3716.580568] LustreError: 111125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.607745] LustreError: 111125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3719.733429] LustreError: 111112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639822 with bad export cookie 4503688899553683246 [ 3719.739369] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3719.745811] LustreError: 111112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3720.152980] Lustre: server umount lustre-MDT0001 complete [ 3730.663613] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3733.098061] Key type lgssc unregistered [ 3733.405858] LNet: 114791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.415829] LNetError: 114791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3734.437561] LNet: Removed LNI 192.168.204.109@tcp [ 3735.364742] Key type .llcrypt unregistered [ 3735.367268] Key type ._llcrypt unregistered [ 3751.330387] Key type ._llcrypt registered [ 3751.332434] Key type .llcrypt registered [ 3751.423135] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3752.487773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3752.568113] alg: No test for adler32 (adler32-zlib) [ 3753.887319] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3754.132715] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3755.872130] Key type lgssc registered [ 3757.204410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.777953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3767.832127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.390937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.884454] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3779.567691] LustreError: 116011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.610270] LustreError: 116011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3784.678888] LustreError: 116012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.393230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.012375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3790.835825] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3799.046089] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3804.992663] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3812.804720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.288080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.352145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3820.149904] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3828.108527] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3841.015673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3841.017283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.029436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.051902] Lustre: Skipped 1 previous similar message [ 3846.134086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.139774] Lustre: Skipped 2 previous similar messages [ 3846.884308] Lustre: server umount lustre-OST0000 complete [ 3852.770542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3852.796942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.829823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.354521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.354943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.400134] Lustre: Skipped 1 previous similar message [ 3858.422167] Lustre: server umount lustre-MDT0000 complete [ 3861.473509] LustreError: 118133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.493033] LustreError: 118133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3862.486533] LustreError: 115992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639965 with bad export cookie 14358367512989557478 [ 3862.493341] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3862.506341] LustreError: 115992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3862.898954] Lustre: server umount lustre-MDT0001 complete [ 3873.261669] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3876.008738] Key type lgssc unregistered [ 3876.312379] LNet: 119544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3876.323151] LNetError: 119544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3876.372690] LNet: Removed LNI 192.168.204.109@tcp [ 3877.143095] Key type .llcrypt unregistered [ 3877.146507] Key type ._llcrypt unregistered [ 3905.961700] Key type ._llcrypt registered [ 3905.963488] Key type .llcrypt registered [ 3906.040497] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3907.065563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3907.111453] alg: No test for adler32 (adler32-zlib) [ 3908.534914] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 3908.806986] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3910.504073] Key type lgssc registered [ 3912.035279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3923.924533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3923.947841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.442407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.886716] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3935.718670] LustreError: 120959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.765494] LustreError: 120959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3940.838430] LustreError: 120960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.859700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.235947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.491208] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3958.796235] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3965.742502] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3973.146245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3973.472184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.505360] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3985.202823] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3996.014465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4006.069277] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4009.763945] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4009.771262] Lustre: Skipped 2 previous similar messages [ 4013.178092] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4013.230540] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4014.563321] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4014.569899] Lustre: Skipped 2 previous similar messages [ 4019.803787] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4019.816367] Lustre: Skipped 1 previous similar message [ 4024.987663] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4024.996821] Lustre: Skipped 2 previous similar messages [ 4026.548814] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.9@tcp 192.168.0.[2,10]@tcp [ 4041.258684] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.9/24@tcp in log lustre-MDT0000 [ 4041.272176] Lustre: Skipped 2 previous similar messages [ 4047.405259] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.9/24@tcp [ 4047.409029] Lustre: Skipped 1 previous similar message [ 4054.496759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4054.507760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.523926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4055.028570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4055.036244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.038436] LustreError: Skipped 1 previous similar message [ 4055.069627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4056.193898] Lustre: server umount lustre-OST0000 complete [ 4062.689170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.696078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4062.698274] Lustre: Skipped 1 previous similar message [ 4066.979456] Lustre: server umount lustre-MDT0000 complete [ 4067.812265] LustreError: 120960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4067.847081] LustreError: 120960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4071.759332] LustreError: 120941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640174 with bad export cookie 8260535944955504579 [ 4071.779755] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.791441] LustreError: 120941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4072.461208] Lustre: server umount lustre-MDT0001 complete [ 4088.048583] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4091.496710] Key type lgssc unregistered [ 4091.904459] LNet: 125802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.916246] LNetError: 125802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.950624] LNet: Removed LNI 192.168.204.109@tcp [ 4093.006470] Key type .llcrypt unregistered [ 4093.009340] Key type ._llcrypt unregistered [ 4125.965157] Key type ._llcrypt registered [ 4125.968486] Key type .llcrypt registered [ 4126.136813] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4128.212700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.256433] alg: No test for adler32 (adler32-zlib) [ 4129.362163] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 4129.648390] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4131.352950] Key type lgssc registered [ 4132.693317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4142.855732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4142.904240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.256696] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4144.270130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.274077] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.9/24@tcp [ 4148.991668] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4154.852103] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.872147] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4158.935547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.359588] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4159.385073] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.9/24@tcp [ 4159.467466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4164.491768] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4173.646200] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.955641] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4189.196630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4189.649834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.748365] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4207.425786] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4214.276417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4229.612166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4229.618063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.635133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4229.635397] Lustre: Skipped 1 previous similar message [ 4229.655132] Lustre: Skipped 1 previous similar message [ 4233.694995] Lustre: server umount lustre-OST0000 complete [ 4239.843257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.846254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.860521] Lustre: Skipped 1 previous similar message [ 4239.879173] Lustre: Skipped 1 previous similar message [ 4244.649475] Lustre: server umount lustre-MDT0000 complete [ 4244.963255] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.988815] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4249.068615] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.104561] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4249.666323] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640352 with bad export cookie 7562210811362203842 [ 4249.676059] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4249.682774] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4250.110525] Lustre: server umount lustre-MDT0001 complete [ 4262.340483] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4265.769236] Key type lgssc unregistered [ 4266.088693] LNet: 130982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4266.096973] LNetError: 130982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4266.110813] LNet: Removed LNI 192.168.204.109@tcp [ 4267.010147] Key type .llcrypt unregistered [ 4267.017522] Key type ._llcrypt unregistered [ 4298.403908] Key type ._llcrypt registered [ 4298.409259] Key type .llcrypt registered [ 4298.495553] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4315.952460] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4316.940224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.972888] alg: No test for adler32 (adler32-zlib) [ 4318.366503] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 4318.711977] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4320.463173] Key type lgssc registered [ 4321.958763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4364.998842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.064371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.328785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4366.358597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4366.430668] Lustre: lustre-MDT0000: new disk, initializing [ 4366.496930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.525568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4373.462301] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4386.842358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.029976] Lustre: 134376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4387.145396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4387.152121] Lustre: Skipped 1 previous similar message [ 4387.248959] Lustre: lustre-MDT0001: new disk, initializing [ 4387.333896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.370765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4387.383246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4393.185100] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4403.816134] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4411.182265] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.840989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.316384] Lustre: lustre-OST0000: new disk, initializing [ 4420.320712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4420.328399] Lustre: 136355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4420.454855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.946156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4427.967723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4428.188198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4428.632844] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4440.082961] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.545873] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4446.937304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4452.930497] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4453.107144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4458.978217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.981829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.991840] LustreError: Skipped 1 previous similar message [ 4459.000886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4459.005105] Lustre: Skipped 1 previous similar message [ 4459.029629] Lustre: Skipped 1 previous similar message [ 4462.058157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4462.072367] Lustre: Skipped 1 previous similar message [ 4467.172915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4467.187930] Lustre: Skipped 1 previous similar message [ 4470.244282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4470.507182] Lustre: server umount lustre-OST0000 complete [ 4477.410792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.412172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.438596] Lustre: Skipped 1 previous similar message [ 4481.679978] Lustre: server umount lustre-MDT0000 complete [ 4486.286759] LustreError: 137697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640589 with bad export cookie 18100976564994405312 [ 4486.291843] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4486.304310] LustreError: 137697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4486.911747] Lustre: server umount lustre-MDT0001 complete [ 4498.377649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.155274] LustreError: 138576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.184154] LustreError: 138576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4499.274251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.559079] LustreError: 138577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.831352] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4509.667291] LustreError: 138576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.811616] LustreError: 138577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.821197] LustreError: 138577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4516.255339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.966246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.215953] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4532.150448] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4538.938862] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4547.685246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.285615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.045273] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4564.661257] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4570.979440] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4572.432902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4577.945499] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4578.122550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4589.025648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4589.316752] Lustre: lustre-OST0001: new disk, initializing [ 4589.334108] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4589.344724] Lustre: 141958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4589.509963] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4598.350402] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4598.968188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4598.988446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4599.132596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4607.809495] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4608.080946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4613.322301] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4613.615828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4620.672558] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f64509a8800.ost_server_uuid 50 [ 4622.683651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f64509a8800.ost_server_uuid in IDLE state after 0 sec [ 4634.630292] Lustre: server umount lustre-OST0001 complete [ 4635.104341] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.120469] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.129787] LustreError: 140400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.135084] Lustre: Skipped 1 previous similar message [ 4635.164450] LustreError: 140400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4639.717457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.727797] LustreError: Skipped 1 previous similar message [ 4639.737177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.759776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.765426] Lustre: Skipped 1 previous similar message [ 4645.329412] Lustre: server umount lustre-OST0000 complete [ 4649.954451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.975085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.988866] Lustre: Skipped 1 previous similar message [ 4649.998746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.007331] Lustre: Skipped 2 previous similar messages [ 4664.287261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4664.722563] Lustre: server umount lustre-MDT0000 complete [ 4665.830733] LustreError: 138577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.855164] LustreError: 138577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4670.002061] LustreError: 138558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640772 with bad export cookie 18100976564994406194 [ 4670.006455] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.020509] LustreError: 138558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4670.790154] Lustre: server umount lustre-MDT0001 complete [ 4697.547102] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4702.140754] Key type lgssc unregistered [ 4702.721095] LNet: 144962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.753708] LNetError: 144962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.779346] LNet: Removed LNI 192.168.204.109@tcp [ 4704.246276] Key type .llcrypt unregistered [ 4704.248813] Key type ._llcrypt unregistered [ 4758.078069] Key type ._llcrypt registered [ 4758.080865] Key type .llcrypt registered [ 4758.214874] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4776.781937] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4777.706590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4777.954891] alg: No test for adler32 (adler32-zlib) [ 4779.048594] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 4779.328632] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4781.079192] Key type lgssc registered [ 4782.424633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4823.876648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4823.896794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.141331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4825.176937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4825.311793] Lustre: lustre-MDT0000: new disk, initializing [ 4825.476919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.512407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4830.500533] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4841.734623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.856101] Lustre: 149134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4841.925258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4841.930362] Lustre: Skipped 1 previous similar message [ 4842.009256] Lustre: lustre-MDT0001: new disk, initializing [ 4842.094662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4842.144387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4842.155690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4847.211656] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4858.557613] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4866.756402] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4875.901745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4876.398350] Lustre: lustre-OST0000: new disk, initializing [ 4876.402299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4876.410949] Lustre: 151106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4876.520835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.011850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4883.019845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4883.166598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4883.685767] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4893.346858] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4899.026576] Lustre: Failing over lustre-OST0000 [ 4899.368823] Lustre: server umount lustre-OST0000 complete [ 4916.192829] Lustre: 147353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641003/real 1788641003] req@000000002adef04a x1875525901520640/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788641019 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4916.228390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.502794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.931446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.977653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.013804] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4919.317500] Lustre: 147356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641006/real 1788641006] req@00000000c2484460 x1875525901521024/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788641022 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4922.036341] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4924.052662] Lustre: 147353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641011/real 1788641011] req@000000008d094575 x1875525901522688/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788641027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4924.089577] Lustre: 147353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4926.743327] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4927.837738] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4927.839506] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4930.759152] Lustre: Failing over lustre-MDT0000 [ 4931.096176] Lustre: server umount lustre-MDT0000 complete [ 4931.252190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.807632] Lustre: 147353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641017/real 1788641017] req@000000002f5c1dac x1875525901523712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788641033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4936.678142] LustreError: 150399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.692453] LustreError: 150399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4938.008710] LustreError: 149141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.795381] LustreError: 149142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.829668] LustreError: 149142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4946.920776] LustreError: 149141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.961245] LustreError: 149141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4947.937755] Lustre: 147354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641034/real 1788641034] req@00000000fe0451e9 x1875525901534720/t0(0) o400->MGC192.168.204.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788641050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4947.996661] Lustre: 147354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4948.014169] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.582439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.063461] LustreError: 149140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.086972] LustreError: 149140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4957.158253] LustreError: 147352:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000f905410 x1875525901540224/t0(0) o250->MGC192.168.204.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4957.489076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.547335] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4958.496791] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4962.805210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4962.808774] Lustre: Skipped 1 previous similar message [ 4962.832152] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4962.879289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4963.131596] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4973.038708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.057621] Lustre: Skipped 2 previous similar messages [ 4973.068359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4973.078138] Lustre: Skipped 1 previous similar message [ 4978.147060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4978.151230] Lustre: Skipped 1 previous similar message [ 4978.514153] Lustre: server umount lustre-OST0000 complete [ 4983.266396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4983.269554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.302821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4983.329020] Lustre: Skipped 2 previous similar messages [ 4988.385873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4988.406402] Lustre: Skipped 2 previous similar messages [ 4989.473494] Lustre: server umount lustre-MDT0000 complete [ 4993.508178] LustreError: 150531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.543264] LustreError: 150531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4994.697692] LustreError: 149127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641097 with bad export cookie 17543788363155959320 [ 4994.703649] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.738893] LustreError: 149127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4995.608188] Lustre: server umount lustre-MDT0001 complete [ 5009.384403] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5012.849265] Key type lgssc unregistered [ 5013.291088] LNet: 154764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5013.306767] LNetError: 154764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5013.333646] LNet: Removed LNI 192.168.204.109@tcp [ 5014.654419] Key type .llcrypt unregistered [ 5014.665069] Key type ._llcrypt unregistered [ 5056.021176] Key type ._llcrypt registered [ 5056.023926] Key type .llcrypt registered [ 5056.139155] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5057.073084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5057.206370] alg: No test for adler32 (adler32-zlib) [ 5058.327734] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 5058.618545] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 5060.312611] Key type lgssc registered [ 5061.279818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5069.257736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5069.287576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.915687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.996684] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5081.072523] LustreError: 156284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.093415] LustreError: 156284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5086.177395] LustreError: 156283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.042646] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5091.042242] LustreError: 156312:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5091.054781] LustreError: 156312:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5091.069336] LustreError: 156312:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5091.597284] Lustre: server umount lustre-MDT0000 complete [ 5108.730540] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5120.539424] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5161.339765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.660776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5161.692649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5161.767482] Lustre: lustre-MDT0000: new disk, initializing [ 5161.889560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.917076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5166.747776] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5177.981214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.180872] Lustre: 159615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5178.258135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5178.262332] Lustre: Skipped 1 previous similar message [ 5178.395491] Lustre: lustre-MDT0001: new disk, initializing [ 5178.602330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.706693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5178.725518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5183.593904] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5193.147283] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.486897] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.308220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.597710] Lustre: lustre-OST0000: new disk, initializing [ 5204.604677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5204.611490] Lustre: 161584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5204.717598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.183678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5210.201731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5210.310296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5211.165536] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5220.992339] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5224.997440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5226.524707] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:48:47 (1788641327) [ 5512.672297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.680534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.693361] Lustre: Skipped 1 previous similar message [ 5517.792988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.808735] Lustre: Skipped 4 previous similar messages [ 5520.782910] Lustre: server umount lustre-MDT0000 complete [ 5522.914213] LustreError: 159627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.939563] LustreError: 159627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5524.785796] LustreError: 160410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641627 with bad export cookie 4088983214621932466 [ 5524.787740] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5524.794514] LustreError: 160410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5528.035869] LustreError: 159622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.040651] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.074814] Lustre: Skipped 1 previous similar message [ 5528.078524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5534.177112] LustreError: 159624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.187459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5534.205246] LustreError: 159624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5534.227871] Lustre: Skipped 1 previous similar message [ 5539.296928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5539.297736] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5539.688165] Lustre: server umount lustre-MDT0001 complete [ 5550.397393] Lustre: server umount lustre-OST0000 complete [ 5560.192342] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5571.240549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5605.806716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.056983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5606.095696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5606.193835] Lustre: lustre-MDT0000: new disk, initializing [ 5606.291036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.327462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5610.867364] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5621.628315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.775151] Lustre: 166259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5621.837704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5621.840976] Lustre: Skipped 1 previous similar message [ 5621.906313] Lustre: lustre-MDT0001: new disk, initializing [ 5622.009430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.040487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5622.044594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5627.176453] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5637.580938] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5644.574240] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5652.123839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5652.467856] Lustre: lustre-OST0000: new disk, initializing [ 5652.480535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5652.499951] Lustre: 168228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5652.592552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.265856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5660.273424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.421710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5660.615541] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5670.982897] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5676.493055] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5676.756667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5682.822307] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5683.067675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5686.269748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.277018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5686.292911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.317083] LustreError: Skipped 1 previous similar message [ 5691.620814] Lustre: server umount lustre-OST0000 complete [ 5696.487645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.514325] Lustre: Skipped 2 previous similar messages [ 5706.736801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5706.752817] Lustre: Skipped 8 previous similar messages [ 5710.303186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5710.563467] Lustre: server umount lustre-MDT0000 complete [ 5711.849518] LustreError: 166267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.884255] LustreError: 166267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5714.923283] LustreError: 166254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641817 with bad export cookie 4088983214622091296 [ 5714.939304] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.354128] Lustre: server umount lustre-MDT0001 complete [ 5724.957127] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:57:05 (1788641825) [ 5745.370566] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5755.865191] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5800.712991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.089950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5801.157682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5801.342571] Lustre: lustre-MDT0000: new disk, initializing [ 5801.482431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.510886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5806.329478] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5817.819284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.977922] Lustre: 172988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5818.362784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5822.228336] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5830.943646] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5837.239772] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5844.091626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5844.315240] Lustre: lustre-OST0000: new disk, initializing [ 5844.318048] Lustre: Skipped 1 previous similar message [ 5844.321979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5844.326846] Lustre: Skipped 2 previous similar messages [ 5844.331549] Lustre: 174968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5850.983402] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5853.252955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5853.258501] Lustre: Skipped 1 previous similar message [ 5853.265894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5853.361386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5861.483537] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.150234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.152489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.164873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.180031] Lustre: Skipped 1 previous similar message [ 5877.391963] Lustre: server umount lustre-OST0000 complete [ 5884.386608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.407312] Lustre: Skipped 2 previous similar messages [ 5888.255447] Lustre: server umount lustre-MDT0000 complete [ 5889.514052] LustreError: 172998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.552325] LustreError: 172998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5892.596500] LustreError: 172981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641995 with bad export cookie 4088983214622092178 [ 5892.607519] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5892.619661] LustreError: 172981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5893.081439] Lustre: server umount lustre-MDT0001 complete [ 5902.308137] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:00:02 (1788642002) [ 5919.184573] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5929.797554] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5967.854900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.051887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5968.074874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5968.140691] Lustre: lustre-MDT0000: new disk, initializing [ 5968.199396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.206327] Lustre: Skipped 2 previous similar messages [ 5968.221342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5972.766792] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5983.824459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.991845] Lustre: 179299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5984.246853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5989.241568] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5998.792859] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6004.654363] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6011.973628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.434218] Lustre: 181278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6018.743211] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6022.745172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6022.911482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6028.379669] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6038.506070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.506916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6038.535305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6038.538971] LustreError: Skipped 2 previous similar messages [ 6038.549668] Lustre: Skipped 3 previous similar messages [ 6041.741957] Lustre: server umount lustre-OST0000 complete [ 6045.664697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6051.743499] Lustre: server umount lustre-MDT0000 complete [ 6053.861687] LustreError: 179313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.873587] LustreError: 179313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6055.202973] LustreError: 179294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642158 with bad export cookie 4088983214622093263 [ 6055.209091] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.215523] LustreError: 179294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6055.599598] Lustre: server umount lustre-MDT0001 complete [ 6065.754308] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 6068.747909] Key type lgssc unregistered [ 6069.057826] LNet: 183133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6069.062947] LNetError: 183133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6069.097098] LNet: Removed LNI 192.168.204.109@tcp [ 6069.914843] Key type .llcrypt unregistered [ 6069.917917] Key type ._llcrypt unregistered [ 6096.538828] Key type ._llcrypt registered [ 6096.541928] Key type .llcrypt registered [ 6096.740867] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 6097.799305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6097.823089] alg: No test for adler32 (adler32-zlib) [ 6098.930147] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6099.119912] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 6100.759234] Key type lgssc registered [ 6101.768509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6109.104270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6109.125511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.520933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.423944] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6120.931133] LustreError: 184547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.955614] LustreError: 184547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6124.797277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.212230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6130.035095] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6138.830497] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6144.998879] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6153.160471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6153.458061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.536028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6161.070147] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6170.372117] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.595819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6191.603421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.611190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.623850] Lustre: Skipped 1 previous similar message [ 6196.475570] Lustre: server umount lustre-OST0000 complete [ 6201.824333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6201.829058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.842540] LustreError: Skipped 1 previous similar message [ 6201.849906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6201.865100] Lustre: Skipped 1 previous similar message [ 6201.885812] Lustre: Skipped 2 previous similar messages [ 6206.946846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.955833] Lustre: Skipped 1 previous similar message [ 6212.066953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.077936] Lustre: Skipped 1 previous similar message [ 6215.135231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6215.282958] Lustre: server umount lustre-MDT0000 complete [ 6217.191040] LustreError: 186898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.224796] LustreError: 186898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6219.600705] LustreError: 184529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642322 with bad export cookie 10216186100038622502 [ 6219.605084] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6219.615826] LustreError: 184529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6220.006122] Lustre: server umount lustre-MDT0001 complete [ 6232.089854] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 6235.238542] Key type lgssc unregistered [ 6235.624084] LNet: 188102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6235.630611] LNetError: 188102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6236.648844] LNet: Removed LNI 192.168.204.109@tcp [ 6237.719904] Key type .llcrypt unregistered [ 6237.721878] Key type ._llcrypt unregistered [ 6263.527225] Key type ._llcrypt registered [ 6263.530368] Key type .llcrypt registered [ 6263.679353] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 6264.793273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6264.828092] alg: No test for adler32 (adler32-zlib) [ 6265.843707] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 6266.061364] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 6267.751179] Key type lgssc registered [ 6269.069165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6276.956063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6276.979406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.774041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6284.762333] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6288.352815] LustreError: 189517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.364785] LustreError: 189517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6293.490043] LustreError: 189516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.085232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6294.489519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6299.729796] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6306.986824] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6313.170449] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6319.861606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.053870] LustreError: 191338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.274364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.403950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6327.395470] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6336.153488] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.164813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.171605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.179197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.193678] Lustre: Skipped 1 previous similar message [ 6344.210316] Lustre: Skipped 1 previous similar message [ 6349.293072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.306604] Lustre: Skipped 2 previous similar messages [ 6350.203695] Lustre: server umount lustre-OST0000 complete [ 6358.045480] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f64509ac000.ost_server_uuid 50 [ 6359.805548] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6376.422597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6376.432218] LustreError: Skipped 1 previous similar message [ 6376.440042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.478125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6376.490043] Lustre: Skipped 1 previous similar message [ 6380.002204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6380.002882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.628716] Lustre: server umount lustre-MDT0000 complete [ 6385.121350] LustreError: 189517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.149500] LustreError: 189517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6385.955545] LustreError: 189498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642488 with bad export cookie 14528705540114424345 [ 6385.978738] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6386.378946] Lustre: server umount lustre-MDT0001 complete [ 6395.471404] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:08:15 (1788642495) [ 6402.264626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.700441] LustreError: 193431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.788536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.906759] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6412.266744] LustreError: 193432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.293381] LustreError: 193432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6416.037507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.595184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6421.644355] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6430.948096] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6437.239867] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6444.392158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6444.727946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.794185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6450.563642] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6457.697814] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6464.329251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.557544] Lustre: lustre-OST0001: new disk, initializing [ 6464.561875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6464.569429] Lustre: 196273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6464.637494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6470.970185] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6471.344735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6471.359244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6471.511044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6480.780733] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6483.947534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.958232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.976625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6483.982780] Lustre: Skipped 1 previous similar message [ 6487.012374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6492.128457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.144802] Lustre: Skipped 5 previous similar messages [ 6498.271156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6498.497596] Lustre: server umount lustre-OST0000 complete [ 6502.187580] LustreError: 195253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6506.084255] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6506.325846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6510.851331] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6511.076814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6527.983906] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6527.989719] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.994874] LustreError: Skipped 1 previous similar message [ 6527.996387] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6527.996405] Lustre: Skipped 3 previous similar messages [ 6528.037256] Lustre: Skipped 2 previous similar messages [ 6531.644111] Lustre: server umount lustre-OST0001 complete [ 6538.215900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.233601] Lustre: Skipped 1 previous similar message [ 6542.133548] Lustre: server umount lustre-MDT0000 complete [ 6543.338058] LustreError: 193431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.361461] LustreError: 193431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 6546.302559] LustreError: 193412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642649 with bad export cookie 14528705540114425416 [ 6546.307583] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6546.319959] LustreError: 193412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6546.783736] Lustre: server umount lustre-MDT0001 complete [ 6575.703512] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:11:16 (1788642676) [ 6582.714834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.249242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6583.280348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6583.560606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.600629] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6598.059079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.168128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6598.200735] Lustre: 199714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.996573] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6609.780423] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6616.949875] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6624.664985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.002611] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6631.538196] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6632.657110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6639.543422] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.871760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.959704] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6654.317895] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6662.946049] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.771383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6668.790399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6668.820561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6668.825499] Lustre: Skipped 4 previous similar messages [ 6672.050800] Lustre: server umount lustre-OST0000 complete [ 6674.203495] LustreError: 201565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.275218] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.294924] LustreError: Skipped 2 previous similar messages [ 6694.573775] Lustre: server umount lustre-OST0001 complete [ 6705.579700] Lustre: server umount lustre-MDT0000 complete [ 6708.971216] LustreError: 201586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642811 with bad export cookie 14528705540114426823 [ 6708.980268] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6708.986414] LustreError: 201586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6709.430784] Lustre: server umount lustre-MDT0001 complete [ 6735.528743] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:13:55 (1788642835) [ 6752.617286] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 6762.872545] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 6807.628655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.947479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6807.973749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6808.047559] Lustre: lustre-MDT0000: new disk, initializing [ 6808.109879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.118460] Lustre: Skipped 3 previous similar messages [ 6808.144192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6812.999160] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6824.559635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6824.713308] Lustre: 207818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6824.800231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6824.810873] Lustre: Skipped 1 previous similar message [ 6824.898550] Lustre: lustre-MDT0001: new disk, initializing [ 6824.904586] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6824.912284] Lustre: Skipped 7 previous similar messages [ 6825.052116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6825.062481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6830.318359] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6840.197729] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6847.179801] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6855.927274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.211258] Lustre: lustre-OST0000: new disk, initializing [ 6856.214993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6856.221035] Lustre: 209790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6863.936756] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6866.572182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6866.595497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6866.829190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6875.668353] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6881.272137] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6881.480886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6886.574121] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6886.935860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6892.517597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6892.523950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.531330] Lustre: Skipped 5 previous similar messages [ 6896.030903] Lustre: server umount lustre-OST0000 complete [ 6905.803966] Lustre: server umount lustre-MDT0000 complete [ 6907.875976] LustreError: 207825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.893465] LustreError: 207825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6909.632459] LustreError: 207812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643012 with bad export cookie 14528705540114428377 [ 6909.644673] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.648261] LustreError: 207812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6909.941609] Lustre: server umount lustre-MDT0001 complete [ 6919.561159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.150211] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6934.175164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.359568] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6946.993750] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6953.373925] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6960.781293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6961.081757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.087040] Lustre: Skipped 4 previous similar messages [ 6967.534729] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 6974.947600] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.992910] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6980.172171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6984.942024] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6985.142996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6989.795316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6989.795835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.806773] LustreError: Skipped 2 previous similar messages [ 6989.821300] Lustre: Skipped 3 previous similar messages [ 6989.833071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.841048] Lustre: Skipped 6 previous similar messages [ 6993.579297] Lustre: server umount lustre-OST0000 complete [ 7001.363264] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7006.756323] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7036.458189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7044.157868] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7051.991375] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7054.190063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7054.220833] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7054.225278] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7054.996949] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7065.917969] Lustre: server umount lustre-OST0000 complete [ 7076.452791] Lustre: server umount lustre-MDT0000 complete [ 7080.617250] LustreError: 211991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643183 with bad export cookie 14528705540114429266 [ 7080.634901] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.635696] LustreError: 211991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7081.069375] Lustre: server umount lustre-MDT0001 complete [ 7088.999685] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:19:49 (1788643189) [ 7095.942525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.482730] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7109.566797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.976833] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7123.861511] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7129.824967] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7136.378219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.121152] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7150.736277] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7155.918772] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7156.242603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7161.587581] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7161.855088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7168.806623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.056706] Lustre: lustre-OST0001: new disk, initializing [ 7169.059769] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7169.064688] Lustre: 221344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7176.245486] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7178.913033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7178.932418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7179.047404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7185.479657] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7190.217766] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7190.487240] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7194.881502] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7195.092328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7199.721334] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7199.729960] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.744288] LustreError: Skipped 3 previous similar messages [ 7199.766762] Lustre: Skipped 6 previous similar messages [ 7202.979512] Lustre: server umount lustre-OST0001 complete [ 7204.840690] LustreError: 221806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.873966] LustreError: 221806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7209.564398] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7209.872678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7215.082870] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7215.314965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7245.407800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.577981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7245.586959] Lustre: Skipped 5 previous similar messages [ 7250.935397] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7250.966052] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7250.979057] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7250.984662] Lustre: Skipped 1 previous similar message [ 7251.855905] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7259.102701] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7260.809150] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7265.112040] Lustre: lustre-OST0001: Not available for connect from 192.168.204.9@tcp (stopping) [ 7265.117819] Lustre: Skipped 7 previous similar messages [ 7268.706227] Lustre: server umount lustre-OST0001 complete [ 7292.417563] Lustre: server umount lustre-OST0000 complete [ 7302.736642] Lustre: server umount lustre-MDT0000 complete [ 7306.851595] LustreError: 224617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643409 with bad export cookie 14528705540114430232 [ 7306.858874] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.874986] LustreError: 224617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7313.322732] Lustre: server umount lustre-MDT0001 complete [ 7337.453102] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:23:58 (1788643438) [ 7343.881960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.144691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7344.174867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7348.530371] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7357.268860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.385658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7357.408982] Lustre: 226591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7361.746213] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7370.308526] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7376.794533] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7384.093856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.405466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7390.952102] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7400.137740] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7412.321357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7412.486597] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7421.242895] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7430.284847] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7432.126938] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7437.466032] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7437.718324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7443.778931] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7443.979984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7450.482371] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f6446e7b000.ost_server_uuid 50 [ 7452.355181] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f6446e7b000.ost_server_uuid in IDLE state after 0 sec [ 7453.674739] Lustre: Permanently deactivating lustre-OST0001 [ 7453.706414] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7458.440918] Lustre: Permanently reactivating lustre-OST0001 [ 7458.454514] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7458.460142] Lustre: Skipped 2 previous similar messages [ 7463.923961] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7463.939354] LustreError: Skipped 3 previous similar messages [ 7463.947708] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.960192] Lustre: Skipped 6 previous similar messages [ 7477.215895] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7477.315924] Lustre: server umount lustre-OST0001 complete [ 7487.758937] Lustre: server umount lustre-OST0000 complete [ 7506.399683] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7506.782047] Lustre: server umount lustre-MDT0000 complete [ 7511.069934] LustreError: 226582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643614 with bad export cookie 14528705540114431520 [ 7511.081152] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7511.084109] LustreError: 226582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7511.549140] Lustre: server umount lustre-MDT0001 complete [ 7540.308177] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:27:20 (1788643640) [ 7548.236581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.710972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7548.757908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7548.772685] Lustre: Skipped 2 previous similar messages [ 7553.971251] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7563.861859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.935288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7563.958244] Lustre: 233292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7568.997582] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7577.473733] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7584.408926] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7591.666667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.869312] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7591.901736] Lustre: Permanently deactivating lustre-OST0000 [ 7591.912401] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7598.876357] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7599.152050] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7607.968947] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7615.978829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.050277] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7622.212695] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7624.929475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7629.980687] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7631.700193] Lustre: Permanently reactivating lustre-OST0000 [ 7631.721569] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7631.729457] Lustre: Skipped 2 previous similar messages [ 7633.323408] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7633.326492] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000f553b511, cur 1788643736 deadline 1788643732 last 1788643702 [ 7633.336663] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7633.357367] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7633.364961] Lustre: Skipped 1 previous similar message [ 7633.401159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7646.303337] Lustre: server umount lustre-OST0001 complete [ 7664.095411] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7664.368935] Lustre: server umount lustre-OST0000 complete [ 7674.614944] Lustre: server umount lustre-MDT0000 complete [ 7678.546094] LustreError: 235163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643781 with bad export cookie 14528705540114433186 [ 7678.555337] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7678.559123] LustreError: 235163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7678.821989] Lustre: server umount lustre-MDT0001 complete [ 7689.540963] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 7692.079161] Key type lgssc unregistered [ 7692.415364] LNet: 238170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7692.423870] LNetError: 238170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7692.446405] LNet: Removed LNI 192.168.204.109@tcp [ 7693.336091] Key type .llcrypt unregistered [ 7693.339100] Key type ._llcrypt unregistered [ 7715.141923] Key type ._llcrypt registered [ 7715.144458] Key type .llcrypt registered [ 7715.255266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 7716.336516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7716.391667] alg: No test for adler32 (adler32-zlib) [ 7717.534357] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 7717.835388] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 7719.511465] Key type lgssc registered [ 7720.500967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.991746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7730.010221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.419151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.203878] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7741.920618] LustreError: 239634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.945954] LustreError: 239634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7743.660549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.805170] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7743.828993] Lustre: 239624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7743.838634] Lustre: Permanently deactivating lustre-MDT0001 [ 7743.848650] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7744.107881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7749.169235] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7749.646437] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7749.656881] LustreError: 239665:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000401:0x1:0x0]: rc = -108 [ 7749.667890] Lustre: 239665:0:(lod_sub_object.c:941:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't open llog [0x240000401:0x1:0x0]: rc = -108 [ 7749.680852] LustreError: 239665:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -108 [ 7759.492971] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.680882] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7774.726234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7775.203554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.235142] LustreError: 241477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.247864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7781.354093] LustreError: 241477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.542121] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7786.465130] LustreError: 241534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.557803] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7790.004969] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7794.659267] LustreError: 241477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.673565] LustreError: 241477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7796.859690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7797.098120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.358986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7802.368504] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7803.116799] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7811.105817] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.216712] Lustre: Permanently reactivating lustre-MDT0001 [ 7814.235708] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7814.242792] Lustre: Skipped 2 previous similar messages [ 7818.558645] Lustre: lustre-MDT0001: Client 28b76d91-6d32-4803-a0c7-ea0911a42361 (at 192.168.204.9@tcp) reconnecting [ 7820.930371] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.943728] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7820.949582] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7820.959466] LustreError: 243259:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7820.963583] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7820.971747] LustreError: 243259:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7820.982843] LustreError: 243259:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.109@tcp: cfg command failed: rc = -108 [ 7820.990804] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7820.990804] [ 7820.998045] LustreError: 239626:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7825.139526] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7826.979563] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7831.885173] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7832.058545] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7836.417674] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7836.704539] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7839.197327] Lustre: Permanently deactivating lustre-MDT0001 [ 7839.210818] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7839.217106] Lustre: Skipped 2 previous similar messages [ 7845.446899] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7851.491125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7851.506602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7851.522423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.539708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.541207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.556856] Lustre: Skipped 1 previous similar message [ 7853.571479] Lustre: Skipped 3 previous similar messages [ 7857.291564] Lustre: server umount lustre-MDT0000 complete [ 7858.660513] LustreError: 240998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7858.674768] LustreError: 240998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7860.829977] LustreError: 239616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643963 with bad export cookie 81960132826563041 [ 7860.841197] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7860.845651] LustreError: 239616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7861.093769] Lustre: server umount lustre-MDT0001 complete [ 7867.582901] Lustre: server umount lustre-OST0000 complete [ 7880.074730] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:33:00 (1788643980) [ 7892.367354] Lustre: server umount lustre-OST0001 complete [ 7899.740105] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 7911.296778] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 7952.170744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.402170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7952.408499] Lustre: Skipped 2 previous similar messages [ 7952.428416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7952.500441] Lustre: lustre-MDT0000: new disk, initializing [ 7952.601867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.635216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7957.186795] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7967.795307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.894440] Lustre: 247894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7967.932705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7967.936222] Lustre: Skipped 1 previous similar message [ 7968.012839] Lustre: lustre-MDT0001: new disk, initializing [ 7968.143303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7968.200850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7968.219171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7972.810266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7982.735609] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7989.031650] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7996.301345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7996.709518] Lustre: lustre-OST0000: new disk, initializing [ 7996.713394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7996.719093] Lustre: 249864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7996.835840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.808516] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8005.703851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8005.713960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8005.834962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8014.473253] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8016.972183] LustreError: 247902:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8026.627300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.782391] Lustre: lustre-OST0001: new disk, initializing [ 8026.789944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8026.796267] Lustre: 250977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8027.016835] LustreError: 247902:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8027.047935] LustreError: 250509:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8032.886617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8032.899434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8033.032294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8033.257819] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8037.063140] LustreError: 250509:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8042.980298] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.544353] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8048.610809] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.621785] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.634063] Lustre: Skipped 1 previous similar message [ 8048.643789] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8048.650910] Lustre: Skipped 1 previous similar message [ 8053.743382] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8058.855795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8058.864208] Lustre: Skipped 2 previous similar messages [ 8061.407492] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8061.613822] Lustre: server umount lustre-OST0001 complete [ 8063.972966] LustreError: 249849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8064.003188] LustreError: 249849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8079.336222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.348240] Lustre: Skipped 1 previous similar message [ 8079.352667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8079.361379] Lustre: Skipped 1 previous similar message [ 8085.218786] Lustre: server umount lustre-OST0000 complete [ 8089.574154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.590365] Lustre: Skipped 1 previous similar message [ 8099.813404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.820776] Lustre: Skipped 7 previous similar messages [ 8103.391227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8104.929938] LustreError: 247906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.949458] LustreError: 247906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8105.649036] Lustre: server umount lustre-MDT0000 complete [ 8109.631406] LustreError: 247886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644212 with bad export cookie 81960132826565855 [ 8109.632048] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8109.651402] LustreError: 247886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8116.184765] Lustre: server umount lustre-MDT0001 complete [ 8130.016904] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8134.009891] Key type lgssc unregistered [ 8134.362341] LNet: 253014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8134.373702] LNetError: 253014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8134.400810] LNet: Removed LNI 192.168.204.109@tcp [ 8135.681098] Key type .llcrypt unregistered [ 8135.684137] Key type ._llcrypt unregistered [ 8186.379844] Key type ._llcrypt registered [ 8186.383403] Key type .llcrypt registered [ 8186.542330] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8187.705597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8187.871059] alg: No test for adler32 (adler32-zlib) [ 8189.018024] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 8189.396767] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8191.137060] Key type lgssc registered [ 8192.412370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.655689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8202.695151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.079032] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8204.116148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8204.285381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.775910] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8219.612249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.701261] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8219.749961] Lustre: 255222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8220.087174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.075733] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8235.192605] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8241.933766] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8249.600572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.994247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8250.292793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.780250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8257.787736] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8268.257095] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8329.184313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.194493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.218434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.697325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.717366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.745634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.972244] Lustre: server umount lustre-OST0000 complete [ 8335.990261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8351.289655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.609066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.003724] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8365.647044] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.032090] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8370.046285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8370.050035] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8370.072920] Lustre: Skipped 1 previous similar message [ 8411.105239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8411.107032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.112479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.127029] LustreError: Skipped 1 previous similar message [ 8411.160766] Lustre: Skipped 1 previous similar message [ 8413.010894] Lustre: server umount lustre-OST0000 complete [ 8419.809877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8419.819403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.843569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8419.846086] Lustre: Skipped 1 previous similar message [ 8423.424511] Lustre: server umount lustre-MDT0000 complete [ 8427.610678] LustreError: 257081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644530 with bad export cookie 15304576525574909027 [ 8427.616884] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.634661] LustreError: 257081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8427.974823] Lustre: server umount lustre-MDT0001 complete [ 8440.734690] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8444.082590] Key type lgssc unregistered [ 8444.414468] LNet: 260646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8444.425562] LNetError: 260646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8444.459657] LNet: Removed LNI 192.168.204.109@tcp [ 8445.428625] Key type .llcrypt unregistered [ 8445.433951] Key type ._llcrypt unregistered [ 8472.086571] Key type ._llcrypt registered [ 8472.088283] Key type .llcrypt registered [ 8472.262878] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8473.278984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8473.305450] alg: No test for adler32 (adler32-zlib) [ 8474.418239] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 8474.699863] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8476.487372] Key type lgssc registered [ 8477.733849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8487.035557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8487.066519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.530929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.202455] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8498.664283] LustreError: 262064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.684319] LustreError: 262064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8503.786391] LustreError: 262065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.313307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.736626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8510.811521] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8520.384161] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8527.641171] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8536.219704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8536.895151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.126371] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8551.351809] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8558.417319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8587.232632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8587.240948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.258629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8589.284327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8589.284558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8589.313194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.069403] Lustre: server umount lustre-OST0000 complete [ 8597.984588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8597.998969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.029096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8598.034623] Lustre: Skipped 1 previous similar message [ 8602.585475] Lustre: server umount lustre-MDT0000 complete [ 8604.643142] LustreError: 262059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8604.665711] LustreError: 262059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8607.049292] LustreError: 262798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644709 with bad export cookie 13902045596210764251 [ 8607.060383] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8607.068509] LustreError: 262798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8607.405810] Lustre: server umount lustre-MDT0001 complete [ 8619.240431] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8621.997510] Key type lgssc unregistered [ 8622.246940] LNet: 266392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8622.257083] LNetError: 266392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8622.281203] LNet: Removed LNI 192.168.204.109@tcp [ 8623.235610] Key type .llcrypt unregistered [ 8623.238396] Key type ._llcrypt unregistered [ 8641.751934] Key type ._llcrypt registered [ 8641.759420] Key type .llcrypt registered [ 8641.925196] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8642.962215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8643.038320] alg: No test for adler32 (adler32-zlib) [ 8644.199762] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 8644.569947] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8646.319348] Key type lgssc registered [ 8647.471878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8656.489445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8656.520584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.055300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.893429] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8667.626413] LustreError: 267612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.649430] LustreError: 267612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8672.738702] LustreError: 267613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8674.554233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.949700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8681.641803] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8690.002694] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8695.903513] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8704.048872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8704.494189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8711.947817] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8719.703516] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8729.082832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8743.667320] Lustre: server umount lustre-OST0000 complete [ 8749.540888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.554679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.555371] Lustre: Skipped 1 previous similar message [ 8749.566874] Lustre: Skipped 1 previous similar message [ 8754.656717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8754.665655] Lustre: Skipped 1 previous similar message [ 8757.794168] Lustre: server umount lustre-MDT0000 complete [ 8759.777818] LustreError: 268539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.790512] LustreError: 268539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8760.799910] Lustre: 266923:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788644847/real 1788644847] req@00000000f9febdb6 x1875529954263040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788644863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8760.819843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.153746] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644864 with bad export cookie 17454409624000898337 [ 8761.159475] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.165481] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8761.453666] Lustre: server umount lustre-MDT0001 complete [ 8774.224842] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8777.048251] Key type lgssc unregistered [ 8777.377916] LNet: 271311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8777.387710] LNetError: 271311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8777.418780] LNet: Removed LNI 192.168.204.109@tcp [ 8778.442577] Key type .llcrypt unregistered [ 8778.445184] Key type ._llcrypt unregistered [ 8804.985558] Key type ._llcrypt registered [ 8804.987198] Key type .llcrypt registered [ 8805.108282] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8806.511929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8806.647372] alg: No test for adler32 (adler32-zlib) [ 8807.745199] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 8807.960221] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8809.623239] Key type lgssc registered [ 8810.998683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8820.028849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8820.041865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.638167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8827.194460] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8831.973106] LustreError: 272725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.001151] LustreError: 272725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8836.698718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.121996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8842.460625] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8850.637633] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8856.353509] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8863.239964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.734877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.809812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8871.020337] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8880.530770] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8912.868240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8912.870898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.876725] LustreError: Skipped 1 previous similar message [ 8912.881797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8912.893311] Lustre: Skipped 1 previous similar message [ 8912.909334] Lustre: Skipped 1 previous similar message [ 8916.468605] Lustre: server umount lustre-OST0000 complete [ 8923.111834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.135788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8923.147848] Lustre: Skipped 1 previous similar message [ 8926.806860] Lustre: server umount lustre-MDT0000 complete [ 8928.239477] LustreError: 272721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.267564] LustreError: 272721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8931.071063] LustreError: 272706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645034 with bad export cookie 1869876487273948621 [ 8931.087754] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.089649] LustreError: 272706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8931.447481] Lustre: server umount lustre-MDT0001 complete [ 8946.387159] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8949.878584] Key type lgssc unregistered [ 8950.263162] LNet: 277032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8950.274882] LNetError: 277032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8950.290841] LNet: Removed LNI 192.168.204.109@tcp [ 8951.340376] Key type .llcrypt unregistered [ 8951.343980] Key type ._llcrypt unregistered [ 8971.105539] Key type ._llcrypt registered [ 8971.109196] Key type .llcrypt registered [ 8971.224562] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8971.976498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8972.039516] alg: No test for adler32 (adler32-zlib) [ 8973.349789] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 8973.532937] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8975.187506] Key type lgssc registered [ 8976.592515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8986.187275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8986.205060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.775745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.708693] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8997.346514] LustreError: 278268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.360188] LustreError: 278268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9002.502067] LustreError: 278270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.799848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.276861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9009.462960] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9017.778108] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9024.635865] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9032.826517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9033.402562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.656762] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9049.535641] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9057.990583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9073.633805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.646336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9073.647749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9073.654890] Lustre: Skipped 1 previous similar message [ 9073.679335] Lustre: Skipped 1 previous similar message [ 9078.046811] Lustre: server umount lustre-OST0000 complete [ 9083.872360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.879653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9083.910468] Lustre: Skipped 1 previous similar message [ 9083.933891] Lustre: Skipped 1 previous similar message [ 9089.000059] LustreError: 278262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9089.021746] LustreError: 278262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9089.161071] Lustre: server umount lustre-MDT0000 complete [ 9093.090743] LustreError: 278272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.117880] LustreError: 278272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9093.736351] LustreError: 278235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645196 with bad export cookie 5833110693308284801 [ 9093.739918] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9093.759243] LustreError: 278235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9094.308851] Lustre: server umount lustre-MDT0001 complete [ 9108.142851] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 9111.316447] Key type lgssc unregistered [ 9111.700816] LNet: 281996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9111.713902] LNetError: 281996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9111.745085] LNet: Removed LNI 192.168.204.109@tcp [ 9112.704972] Key type .llcrypt unregistered [ 9112.710632] Key type ._llcrypt unregistered [ 9126.376123] Key type ._llcrypt registered [ 9126.387497] Key type .llcrypt registered [ 9126.532534] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9147.992673] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9166.956373] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9168.640729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9168.685041] alg: No test for adler32 (adler32-zlib) [ 9169.819600] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 9170.097712] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 9171.759501] Key type lgssc registered [ 9173.174792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9212.961495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9213.015234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.360238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9214.426935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9214.514301] Lustre: lustre-MDT0000: new disk, initializing [ 9214.597525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.642693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9220.781498] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9233.007885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.140253] Lustre: 285548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9233.192074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9233.195488] Lustre: Skipped 1 previous similar message [ 9233.276588] Lustre: lustre-MDT0001: new disk, initializing [ 9233.400730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.437429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9233.449635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9238.771414] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9249.098077] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9254.384591] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9261.426443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.787587] Lustre: lustre-OST0000: new disk, initializing [ 9261.797891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9261.810569] Lustre: 287516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9261.921751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.024804] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9269.350922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9269.369847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9269.488603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9278.566863] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9283.876120] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9284.092515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9289.445588] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9289.694162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9295.330199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.330508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9295.344510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9295.354287] LustreError: Skipped 1 previous similar message [ 9295.374209] Lustre: Skipped 1 previous similar message [ 9297.744159] Lustre: server umount lustre-OST0000 complete [ 9305.057810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9305.071760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.095240] Lustre: Skipped 1 previous similar message [ 9305.109499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9310.688533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9310.695189] Lustre: Skipped 2 previous similar messages [ 9315.816649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9315.822593] Lustre: Skipped 1 previous similar message [ 9316.320468] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9316.585326] Lustre: server umount lustre-MDT0000 complete [ 9321.000988] LustreError: 285541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645423 with bad export cookie 15618984501023908519 [ 9321.001961] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.012961] LustreError: 285541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9321.344463] Lustre: server umount lustre-MDT0001 complete [ 9331.156780] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:57:11 (1788645431) [ 9338.365950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.952910] LustreError: 289930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.969955] LustreError: 289930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9339.031371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.678786] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9344.484825] LustreError: 289931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.510862] LustreError: 289931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9348.580178] LustreError: 289930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.699118] LustreError: 289931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.559735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.134915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9360.672424] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9369.137757] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9376.015816] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9383.439183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.800029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.247498] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9400.212959] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9409.511951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9409.512594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.527290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9409.539749] Lustre: Skipped 2 previous similar messages [ 9415.745991] Lustre: server umount lustre-OST0000 complete [ 9421.794696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9421.807470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.829809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.835335] Lustre: Skipped 3 previous similar messages [ 9427.602587] Lustre: server umount lustre-MDT0000 complete [ 9429.987783] LustreError: 291826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.014946] LustreError: 291826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9434.748987] LustreError: 289911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645537 with bad export cookie 15618984501023909401 [ 9434.780642] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.781805] LustreError: 289911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9435.152458] Lustre: server umount lustre-MDT0001 complete [ 9454.863975] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 9459.464363] Key type lgssc unregistered [ 9459.894870] LNet: 293488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9459.924784] LNetError: 293488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9459.978568] LNet: Removed LNI 192.168.204.109@tcp [ 9460.907125] Key type .llcrypt unregistered [ 9460.917495] Key type ._llcrypt unregistered [ 9492.468189] Key type ._llcrypt registered [ 9492.470498] Key type .llcrypt registered [ 9492.573331] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9508.632768] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9509.874382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9509.932545] alg: No test for adler32 (adler32-zlib) [ 9511.107499] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 9511.343813] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 9512.991453] Key type lgssc registered [ 9514.245824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9567.612417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9567.642161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9568.871771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9568.913626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9569.012529] Lustre: lustre-MDT0000: new disk, initializing [ 9569.163077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9569.224132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9574.548484] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9586.529536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.750302] Lustre: 297326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9586.839307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9586.846363] Lustre: Skipped 1 previous similar message [ 9586.947201] Lustre: lustre-MDT0001: new disk, initializing [ 9587.055725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9587.117401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9587.131097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9593.405181] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9604.641030] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9610.780413] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9618.525890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.750730] Lustre: lustre-OST2710: new disk, initializing [ 9618.758437] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9618.767703] Lustre: 299294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9618.873207] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9625.264964] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9628.796777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9628.822629] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9629.012620] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9636.488295] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9644.031091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.391503] Lustre: lustre-OST03e8: new disk, initializing [ 9644.403332] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9644.413325] Lustre: 300360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9644.638745] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9649.768772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9649.779126] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9649.902108] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9652.468216] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9661.075266] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9669.274829] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9669.633409] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9675.230761] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9675.523628] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9680.682601] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9680.895424] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9685.892380] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9686.172615] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9694.691921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9694.703076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.723342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9696.224102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9696.225697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.248750] Lustre: Skipped 1 previous similar message [ 9697.824801] Lustre: server umount lustre-MDT0000 complete [ 9701.560506] LustreError: 297317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645804 with bad export cookie 40015696445582950 [ 9701.569069] LustreError: 297317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9701.572509] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9701.905966] Lustre: server umount lustre-MDT0001 complete [ 9715.938617] Lustre: server umount lustre-OST2710 complete [ 9732.333378] Lustre: server umount lustre-OST03e8 complete [ 9753.489763] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9768.074819] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9809.144920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.464746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9809.494771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9809.625968] Lustre: lustre-MDT0000: new disk, initializing [ 9809.718794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.737543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9814.665735] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9825.853402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.034228] Lustre: 305609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9826.096741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9826.099259] Lustre: Skipped 1 previous similar message [ 9826.176517] Lustre: lustre-MDT0001: new disk, initializing [ 9826.275432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9826.306198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9826.330724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9831.783403] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9842.842575] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9850.063620] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9857.754547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9858.067299] Lustre: lustre-OST0000: new disk, initializing [ 9858.070856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9858.089956] Lustre: 307585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9858.194134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9865.705466] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9866.325933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9866.332355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9866.535844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9877.208920] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9883.514855] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9883.714988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9890.082395] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9890.384516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9897.442726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.445485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9897.462073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9897.462083] Lustre: Skipped 3 previous similar messages [ 9897.473828] Lustre: Skipped 2 previous similar messages [ 9898.827891] Lustre: server umount lustre-OST0000 complete [ 9907.712380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.717221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9907.729064] Lustre: Skipped 1 previous similar message [ 9907.736230] Lustre: Skipped 1 previous similar message [ 9909.377411] Lustre: server umount lustre-MDT0000 complete [ 9912.800218] LustreError: 306772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.823643] LustreError: 306772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9913.586747] LustreError: 307583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646016 with bad export cookie 40015696445584644 [ 9913.593738] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.603893] LustreError: 307583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9914.039441] Lustre: server umount lustre-MDT0001 complete [ 9923.110912] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:07:03 (1788646023) [ 9924.592836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9926.346719] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:07:06 (1788646026) [ 9930.502725] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9962.157961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.624947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9962.640896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9962.897103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.684486] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9983.002703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.175464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9983.209444] Lustre: 311443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9988.652300] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10012.334678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10012.537876] Lustre: Denying initial registration attempt from nid 192.168.204.109@tcp, specified as failover [10012.547265] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10012.558869] LustreError: 312898:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [10012.563148] LustreError: 312898:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10012.567372] LustreError: 312898:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10012.636274] Lustre: server umount lustre-OST0000 complete [10012.639259] LustreError: 312898:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10019.296173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10019.305576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.320783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10025.437937] Lustre: server umount lustre-MDT0000 complete [10025.448644] LustreError: 311450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.467982] LustreError: 311450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10028.839979] LustreError: 311434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646131 with bad export cookie 40015696445585526 [10028.849247] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10028.860980] LustreError: 311434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10029.162309] Lustre: server umount lustre-MDT0001 complete [10042.684893] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [10046.088632] Key type lgssc unregistered [10046.441542] LNet: 314018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10046.450588] LNetError: 314018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10047.472951] LNet: Removed LNI 192.168.204.109@tcp [10048.485833] Key type .llcrypt unregistered [10048.488327] Key type ._llcrypt unregistered [10061.997422] Key type ._llcrypt registered [10062.003182] Key type .llcrypt registered [10062.110591] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10063.422868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10063.799382] alg: No test for adler32 (adler32-zlib) [10064.886983] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [10065.146366] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [10066.927205] Key type lgssc registered [10068.199901] Lustre: Echo OBD driver; http://www.lustre.org/ [10119.354865] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10129.605736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10129.622975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.854807] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10130.908195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10131.049090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10135.291325] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10144.631499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.695689] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10144.714260] Lustre: 316781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10145.020909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10149.551763] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10157.411271] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10163.180153] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10183.089983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.293422] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10183.480379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10191.777155] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10201.186777] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10204.641543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10204.652449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.667265] LustreError: Skipped 1 previous similar message [10204.669617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10204.674496] Lustre: Skipped 1 previous similar message [10204.725789] Lustre: Skipped 1 previous similar message [10209.564411] Lustre: server umount lustre-OST0000 complete [10215.129531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.139732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.148599] Lustre: Skipped 1 previous similar message [10215.157606] Lustre: Skipped 1 previous similar message [10220.512445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10220.522798] Lustre: Skipped 1 previous similar message [10221.466028] Lustre: server umount lustre-MDT0000 complete [10226.147296] LustreError: 318649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646329 with bad export cookie 16523143799756187704 [10226.149872] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10226.167210] LustreError: 318649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10226.607730] Lustre: server umount lustre-MDT0001 complete [10237.924258] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [10241.298407] Key type lgssc unregistered [10241.716445] LNet: 320355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10241.745850] LNetError: 320355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10241.778983] LNet: Removed LNI 192.168.204.109@tcp [10242.993951] Key type .llcrypt unregistered [10242.996347] Key type ._llcrypt unregistered [10273.105784] Key type ._llcrypt registered [10273.111311] Key type .llcrypt registered [10273.369651] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10274.705623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10274.729595] alg: No test for adler32 (adler32-zlib) [10276.128736] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [10276.631409] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [10278.458073] Key type lgssc registered [10279.888563] Lustre: Echo OBD driver; http://www.lustre.org/ [10288.646366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10288.677436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.237215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.506513] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10300.384557] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.402236] LustreError: 321770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10304.949489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10305.378827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10309.416504] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10316.460641] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10321.744234] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10330.221521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10330.684081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10336.273933] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10344.375310] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10360.305257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.331841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10360.356732] Lustre: Skipped 2 previous similar messages [10362.729789] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [10365.411062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.426901] Lustre: Skipped 2 previous similar messages [10366.495964] Lustre: server umount lustre-MDT0000 complete [10367.760787] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.532427] LustreError: 323203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.556705] LustreError: 323203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10372.110928] LustreError: 321749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646475 with bad export cookie 825618165046146174 [10372.128697] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10372.139779] LustreError: 321749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10372.883154] LustreError: 321766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.942426] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [10375.659497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10375.672925] Lustre: Skipped 2 previous similar messages [10377.697252] LustreError: 322817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10377.700331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10377.724218] LustreError: 322817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10377.740922] Lustre: Skipped 1 previous similar message [10378.865802] Lustre: server umount lustre-MDT0001 complete [10387.083423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10396.030121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10397.166900] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb752f58c7af9206 [10397.193528] Lustre: MGC192.168.204.109@tcp: Connection restored to 0@lo (at 0@lo) [10397.578463] LustreError: 323603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.604173] LustreError: 323603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10397.677179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10397.762944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10401.679814] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10407.913499] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10410.269294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10410.688239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10414.942375] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10416.111900] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10422.635880] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10424.136630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10429.625507] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10431.569267] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10438.624994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10438.632659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.657684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10438.661928] Lustre: Skipped 1 previous similar message [10441.709680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10441.711646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10442.858990] Lustre: server umount lustre-OST0000 complete [10446.305163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10446.313660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.592901] Lustre: server umount lustre-MDT0000 complete [10457.056486] LustreError: 325390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.071869] LustreError: 325390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10458.023914] LustreError: 321749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646560 with bad export cookie 825618165046153734 [10458.028120] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10458.031746] LustreError: 321749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10458.559189] Lustre: server umount lustre-MDT0001 complete [10469.574800] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [10472.453657] Key type lgssc unregistered [10472.746346] LNet: 328057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10472.755206] LNetError: 328057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10472.770539] LNet: Removed LNI 192.168.204.109@tcp [10473.674763] Key type .llcrypt unregistered [10473.676972] Key type ._llcrypt unregistered [10499.946578] Key type ._llcrypt registered [10499.949158] Key type .llcrypt registered [10500.046388] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10501.006763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10501.026237] alg: No test for adler32 (adler32-zlib) [10502.029634] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [10502.216600] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [10503.847214] Key type lgssc registered [10504.745871] Lustre: Echo OBD driver; http://www.lustre.org/ [10512.324055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10512.373570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.831846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.009884] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10523.105829] LustreError: 329471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10523.124025] LustreError: 329471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10527.834191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.314643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10533.038504] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10540.600580] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10546.387905] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10553.379576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.385935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.393675] Lustre: Skipped 1 previous similar message [10553.413804] Lustre: Skipped 1 previous similar message [10555.674477] Lustre: server umount lustre-MDT0000 complete [10558.433428] LustreError: 329466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.477677] LustreError: 329466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10559.912588] LustreError: 329451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646662 with bad export cookie 7685028976721836305 [10559.925348] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10559.928753] LustreError: 329451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10560.371081] Lustre: server umount lustre-MDT0001 complete [10573.411826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.662858] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10573.686509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10573.899120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10578.207408] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10586.823746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.899162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10586.927165] Lustre: 332057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10587.146819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10591.492533] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10598.263831] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10602.802210] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10612.523129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.784742] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10612.804216] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10612.816727] LustreError: 333942:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10612.835468] LustreError: 333942:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10612.848274] LustreError: 333942:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10612.953293] Lustre: server umount lustre-OST0000 complete [10612.957611] LustreError: 333942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10623.406218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.598432] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10623.811134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10630.091370] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10632.882788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10638.479840] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10646.449511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10646.557104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10646.620982] Lustre: lustre-OST0001: new disk, initializing [10646.624782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10646.630601] Lustre: 335487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10646.723392] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10653.179432] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10654.342972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10654.366111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10654.589104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10661.969538] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10670.070794] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10670.075158] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.100114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10671.680443] Lustre: server umount lustre-OST0001 complete [10675.182512] LustreError: 334407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10675.199546] LustreError: 334407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10678.753069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10678.770918] LustreError: Skipped 1 previous similar message [10678.779783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.803185] Lustre: Skipped 1 previous similar message [10678.815170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10678.824220] Lustre: Skipped 1 previous similar message [10680.298269] LustreError: 334406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.317255] LustreError: 334406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10682.080274] Lustre: server umount lustre-OST0000 complete [10689.506445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10689.513092] LustreError: Skipped 1 previous similar message [10689.516849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10689.536279] Lustre: Skipped 1 previous similar message [10689.540909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10689.545945] Lustre: Skipped 3 previous similar messages [10692.858768] Lustre: server umount lustre-MDT0000 complete [10695.650439] LustreError: 332070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.668527] LustreError: 332070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10696.618824] LustreError: 332051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646799 with bad export cookie 7685028976721836858 [10696.628627] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.632349] LustreError: 332051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10697.043406] Lustre: server umount lustre-MDT0001 complete [10707.821682] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [10710.702603] Key type lgssc unregistered [10711.028758] LNet: 337446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10711.038266] LNetError: 337446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10711.059572] LNet: Removed LNI 192.168.204.109@tcp [10712.071247] Key type .llcrypt unregistered [10712.077867] Key type ._llcrypt unregistered [10788.657432] Key type ._llcrypt registered [10788.659764] Key type .llcrypt registered [10788.755915] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [10805.692419] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10807.038175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10807.279405] alg: No test for adler32 (adler32-zlib) [10808.458575] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [10808.768617] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [10810.479188] Key type lgssc registered [10811.594395] Lustre: Echo OBD driver; http://www.lustre.org/ [10842.873629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10842.893649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.121740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10844.161638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10844.249615] Lustre: lustre-MDT0000: new disk, initializing [10844.359401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.381416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10848.431668] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10858.377952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10858.471853] Lustre: 342378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10858.517710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10858.521620] Lustre: Skipped 1 previous similar message [10858.622665] Lustre: lustre-MDT0001: new disk, initializing [10858.729637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10858.779115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10858.809760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10862.873975] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10872.543587] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10878.027340] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10884.782835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10885.116862] Lustre: lustre-OST0000: new disk, initializing [10885.125281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10885.133669] Lustre: 344347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10885.229829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10892.272777] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10892.453872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10892.466204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10892.664318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10901.628925] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10906.970673] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10907.233620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10911.852934] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10912.114313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10918.380722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10918.381401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.387824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10918.392826] LustreError: Skipped 1 previous similar message [10918.431521] Lustre: Skipped 1 previous similar message [10920.369843] Lustre: server umount lustre-OST0000 complete [10925.539411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10925.547300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10925.556028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10925.560973] Lustre: Skipped 1 previous similar message [10928.609805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.610614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10928.641583] Lustre: Skipped 1 previous similar message [10933.732164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10933.736954] Lustre: Skipped 1 previous similar message [10938.337438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10938.669703] Lustre: server umount lustre-MDT0000 complete [10942.207615] LustreError: 342372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647045 with bad export cookie 5927569001470331169 [10942.214860] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10942.220912] LustreError: 342372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10942.529926] Lustre: server umount lustre-MDT0001 complete [10950.659876] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:24:11 (1788647051) [10958.505565] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:24:18 (1788647058) [10967.757067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.152250] LustreError: 347101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.170359] LustreError: 347101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10968.225902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10973.102736] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10973.674793] LustreError: 347102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.795810] LustreError: 347101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10982.822690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.159401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10988.172275] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10996.383035] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11003.462528] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11011.158661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11011.425454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11018.243185] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11025.607918] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11037.715657] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11039.789691] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11041.289902] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11043.040076] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11044.787273] Lustre: Failing over lustre-MDT0000 [11044.835080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11044.851744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11044.875454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11045.075236] Lustre: server umount lustre-MDT0000 complete [11048.744162] LustreError: 347098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.442302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.460825] Lustre: Skipped 1 previous similar message [11053.845495] LustreError: 347097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.869039] LustreError: 347097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11063.593877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.819859] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11064.092782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11064.143821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11068.817448] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11069.216746] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11069.428616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11069.478861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11069.519105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11077.646975] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11079.055533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11082.233677] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11083.970983] Lustre: Failing over lustre-MDT0000 [11084.235279] Lustre: server umount lustre-MDT0000 complete [11084.594610] LustreError: 347097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.613278] LustreError: 347097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11084.768643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11084.773621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11084.804594] Lustre: Skipped 1 previous similar message [11100.134377] Lustre: 340602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788647187/real 1788647187] req@000000008cc7680a x1875532223438208/t0(0) o400->MGC192.168.204.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788647203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11100.161153] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.995583] LustreError: 349047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.022754] LustreError: 349047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11102.736760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.379715] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5242f499ea4193b6 [11110.401850] Lustre: MGC192.168.204.109@tcp: Connection restored to 0@lo (at 0@lo) [11110.412900] Lustre: Skipped 2 previous similar messages [11110.721780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11110.787480] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11111.184409] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11115.032636] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11116.016746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11116.052615] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11116.118819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11124.787320] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11126.423654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11136.483141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.485284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11136.485404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11136.509848] Lustre: Skipped 2 previous similar messages [11138.388690] Lustre: server umount lustre-MDT0000 complete [11141.601287] LustreError: 347101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.628366] LustreError: 347101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11142.997902] LustreError: 347081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647245 with bad export cookie 5927569001470333878 [11143.002771] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11143.011522] LustreError: 347081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11143.414335] Lustre: server umount lustre-MDT0001 complete [11158.884142] Lustre: server umount lustre-OST0000 complete [11175.889802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11176.518202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11181.889835] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11190.069446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.433074] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11201.005284] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11206.555648] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11213.010087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11219.391219] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11221.613199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11227.351708] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11234.635277] Lustre: server umount lustre-OST0000 complete [11241.441556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11241.453140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.474939] Lustre: Skipped 1 previous similar message [11241.480080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11241.487433] Lustre: Skipped 2 previous similar messages [11248.686821] Lustre: server umount lustre-MDT0000 complete [11251.551447] Lustre: 340605:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788647338/real 1788647338] req@0000000035051e88 x1875532223489024/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788647354 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11251.581151] LustreError: 354517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.607482] LustreError: 354517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11252.603342] LustreError: 355309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647355 with bad export cookie 5927569001470334634 [11252.619679] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11252.621360] LustreError: 355309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11253.097232] Lustre: server umount lustre-MDT0001 complete [11264.327256] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [11267.196787] Key type lgssc unregistered [11267.478049] LNet: 356997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11267.482734] LNetError: 356997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11267.496687] LNet: Removed LNI 192.168.204.109@tcp [11268.369180] Key type .llcrypt unregistered [11268.372503] Key type ._llcrypt unregistered [11298.954494] Key type ._llcrypt registered [11298.959503] Key type .llcrypt registered [11299.126942] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11300.529070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11300.569115] alg: No test for adler32 (adler32-zlib) [11301.697305] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [11301.947660] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [11303.615642] Key type lgssc registered [11304.842714] Lustre: Echo OBD driver; http://www.lustre.org/ [11311.357979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11311.378716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.694947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11316.377575] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11322.852056] LustreError: 358514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.873137] LustreError: 358514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11324.275156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.574490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11328.537458] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11335.628264] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11340.802950] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11347.604524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11347.976153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11354.520503] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11357.261732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11363.802878] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11375.588178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11375.593650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.611718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11377.478309] Lustre: server umount lustre-MDT0000 complete [11377.647107] LustreError: 358514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.647956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11377.678433] Lustre: Skipped 1 previous similar message [11380.985646] LustreError: 358493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647483 with bad export cookie 6970969692077844674 [11380.992271] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11380.995081] LustreError: 358493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11381.404747] Lustre: server umount lustre-MDT0001 complete [11396.314475] Lustre: server umount lustre-OST0000 complete [11409.685276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.167288] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11410.184037] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11410.235456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.513695] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11415.527476] LustreError: 362227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.647516] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.751987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.156914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11427.429343] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11434.216540] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11440.086729] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11447.130695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11447.599054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11454.095989] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11455.800104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11461.179395] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11474.403726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11474.426963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.474542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.453659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.458517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.485741] Lustre: Skipped 1 previous similar message [11476.502223] Lustre: Skipped 2 previous similar messages [11479.980352] Lustre: server umount lustre-MDT0000 complete [11481.578250] LustreError: 362221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.599553] LustreError: 362221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11483.966730] LustreError: 362960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647586 with bad export cookie 6970969692077845787 [11483.974027] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.984923] LustreError: 362960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11484.514665] Lustre: server umount lustre-MDT0001 complete [11497.398984] Lustre: server umount lustre-OST0000 complete [11511.603419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11512.037578] LustreError: 366133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11512.123342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11516.021383] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11524.428798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.035756] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11536.311318] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11541.874590] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11549.712974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.940503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.952902] Lustre: Skipped 1 previous similar message [11556.307133] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11557.107795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11563.639720] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11569.826647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11569.836941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11570.929148] Lustre: server umount lustre-OST0000 complete [11575.270927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11575.275148] Lustre: Skipped 1 previous similar message [11581.475802] Lustre: server umount lustre-MDT0000 complete [11585.506455] LustreError: 366134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11585.517373] LustreError: 366134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11585.987896] LustreError: 366869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647688 with bad export cookie 6970969692077846809 [11585.990452] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11586.011077] LustreError: 366869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11586.509619] Lustre: server umount lustre-MDT0001 complete [11598.660787] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [11601.588671] Key type lgssc unregistered [11601.918776] LNet: 369662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11601.931467] LNetError: 369662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11602.982257] LNet: Removed LNI 192.168.204.109@tcp [11603.853499] Key type .llcrypt unregistered [11603.857307] Key type ._llcrypt unregistered [11629.450337] Key type ._llcrypt registered [11629.455446] Key type .llcrypt registered [11629.602614] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11631.240704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11631.382749] alg: No test for adler32 (adler32-zlib) [11632.607281] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [11633.083423] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [11634.839227] Key type lgssc registered [11636.429206] Lustre: Echo OBD driver; http://www.lustre.org/ [11647.204635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11647.253648] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11647.281993] LustreError: 371102:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11647.311475] LustreError: 371102:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11647.320931] LustreError: 371102:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11647.331130] LustreError: 371102:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11647.341437] LustreError: 371102:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11657.250434] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.257525] LustreError: 371500:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11657.277374] LustreError: 371500:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11657.287304] LustreError: 371500:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11657.294754] LustreError: 371500:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11657.308442] LustreError: 371500:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11683.092231] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [11686.245623] Key type lgssc unregistered [11686.559691] LNet: 372503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11686.565248] LNetError: 372503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11686.578740] LNet: Removed LNI 192.168.204.109@tcp [11687.675852] Key type .llcrypt unregistered [11687.679259] Key type ._llcrypt unregistered [11713.457324] Key type ._llcrypt registered [11713.459229] Key type .llcrypt registered [11713.625338] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [11734.756883] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11735.683408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11735.803227] alg: No test for adler32 (adler32-zlib) [11737.125782] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [11737.439795] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [11739.167253] Key type lgssc registered [11740.551832] Lustre: Echo OBD driver; http://www.lustre.org/ [11781.862451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11781.888285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11783.131840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11783.184158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11783.295610] Lustre: lustre-MDT0000: new disk, initializing [11783.401305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11783.443781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11789.485108] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11802.009748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.203080] Lustre: 375659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11802.262962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11802.266815] Lustre: Skipped 1 previous similar message [11802.386461] Lustre: lustre-MDT0001: new disk, initializing [11802.558885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11802.608392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11802.617926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11808.326863] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11820.068305] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11828.250603] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11835.758247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11836.167825] Lustre: lustre-OST0000: new disk, initializing [11836.177214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11836.194424] Lustre: 377632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11836.329896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11843.879969] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11845.710260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11845.719694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11845.842275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11854.817682] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11860.846757] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11861.035542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11865.802908] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11865.981571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11871.714832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11871.721795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.735049] LustreError: Skipped 1 previous similar message [11871.740194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11871.767685] Lustre: Skipped 1 previous similar message [11874.955224] Lustre: server umount lustre-OST0000 complete [11881.957448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11881.984348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11881.995415] Lustre: Skipped 2 previous similar messages [11886.123955] Lustre: server umount lustre-MDT0000 complete [11892.753106] LustreError: 375650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647995 with bad export cookie 6243063299814471054 [11892.766354] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11892.778135] LustreError: 375650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11893.353653] Lustre: server umount lustre-MDT0001 complete [11908.427204] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11911.283865] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:40:10 (1788648010) [11920.481904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.017837] LustreError: 380093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.043735] LustreError: 380093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11921.109862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11926.006242] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11926.499465] LustreError: 380094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.619452] LustreError: 380093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.436402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.803191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.653207] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11952.549494] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11961.411849] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11970.533213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11971.136715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11979.718121] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11988.631504] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11997.267936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11997.431582] Lustre: lustre-OST0001: new disk, initializing [11997.434867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11997.441434] Lustre: 382934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11997.556850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12005.549933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12005.559152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12005.665458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12005.806622] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12018.916310] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12025.658967] Lustre: lustre-OST0001: Not available for connect from 192.168.204.9@tcp (stopping) [12026.348033] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.350166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12026.369321] Lustre: Skipped 2 previous similar messages [12029.273825] Lustre: server umount lustre-OST0001 complete [12030.754905] LustreError: 381916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.595537] LustreError: 381916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.628089] LustreError: 381916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12046.834939] LustreError: 381918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12046.878137] LustreError: 381918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12047.840761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12047.845275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12047.861028] LustreError: Skipped 1 previous similar message [12047.863649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12047.881032] Lustre: Skipped 1 previous similar message [12047.902085] Lustre: Skipped 3 previous similar messages [12051.956923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12051.972456] Lustre: Skipped 1 previous similar message [12053.803781] Lustre: server umount lustre-OST0000 complete [12059.618075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12059.627742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12062.176442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12062.199770] Lustre: Skipped 2 previous similar messages [12065.186729] Lustre: server umount lustre-MDT0000 complete [12067.305386] LustreError: 380088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.341297] LustreError: 380088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12071.553516] LustreError: 380074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648174 with bad export cookie 6243063299814471936 [12071.575276] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12071.593926] LustreError: 380074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12078.320525] Lustre: server umount lustre-MDT0001 complete [12092.021505] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12095.823874] Key type lgssc unregistered [12096.139270] LNet: 384922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12096.151586] LNetError: 384922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12096.162969] LNet: Removed LNI 192.168.204.109@tcp [12097.208361] Key type .llcrypt unregistered [12097.210115] Key type ._llcrypt unregistered [12157.032342] Key type ._llcrypt registered [12157.037377] Key type .llcrypt registered [12157.158521] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12158.357961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12158.383468] alg: No test for adler32 (adler32-zlib) [12159.485827] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [12159.827313] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12161.487217] Key type lgssc registered [12162.763433] Lustre: Echo OBD driver; http://www.lustre.org/ [12170.995302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12171.013475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.264367] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12172.299819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12172.484727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.302301] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12186.209550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12186.301185] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12186.341924] Lustre: 387718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12186.574653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12190.931283] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12198.248356] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12204.379588] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12211.688192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.706151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12211.719580] Lustre: Skipped 1 previous similar message [12213.802060] Lustre: server umount lustre-MDT0000 complete [12218.876798] LustreError: 387710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648321 with bad export cookie 1146513010974057931 [12218.894029] LustreError: 387710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12218.895824] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12219.453422] Lustre: server umount lustre-MDT0001 complete [12230.974391] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:45:31 (1788648331) [12238.347964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.807053] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.827548] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12238.884808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.937343] LustreError: 390235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.270635] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12249.069771] LustreError: 390234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.933391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12254.248362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.437646] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12266.909975] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12271.821792] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12278.255129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12278.459289] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12278.682807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12284.081661] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12290.876157] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12299.046151] Lustre: Permanently deactivating lustre-OST0000 [12299.070588] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12304.942762] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12306.459516] LustreError: 393174:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12306.469841] LustreError: 393174:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12306.478034] LustreError: 393174:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12309.204385] LustreError: 393223:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12309.214275] LustreError: 393223:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12317.274209] Lustre: server umount lustre-OST0000 complete [12320.248412] LustreError: 393472:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12320.256172] LustreError: 393472:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12325.861229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12325.876635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.892120] Lustre: Skipped 1 previous similar message [12325.904763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12326.882511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.003919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12332.010853] Lustre: Skipped 3 previous similar messages [12333.099368] Lustre: server umount lustre-MDT0000 complete [12336.540173] LustreError: 390215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648439 with bad export cookie 1146513010974058512 [12336.546172] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12336.550463] LustreError: 390215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12347.070946] Lustre: server umount lustre-MDT0001 complete [12355.651965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12360.761255] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12363.564818] LustreError: 394634:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12369.180732] Lustre: 394974:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.109@tcp [12371.454267] Lustre: server umount MGS complete [12383.437431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.011877] LustreError: 395698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12384.050686] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12384.061734] Lustre: Skipped 1 previous similar message [12384.124553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12388.479188] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12389.345505] LustreError: 395699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12399.151617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12399.452757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12404.153482] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12411.054293] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12416.668582] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12418.497616] Lustre: Permanently reactivating lustre-OST0000 [12418.514790] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12418.523109] Lustre: Skipped 2 previous similar messages [12446.692305] LustreError: 397490:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12454.370522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12454.685719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12454.692886] Lustre: Skipped 1 previous similar message [12460.756515] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12468.790558] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12477.408308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12477.430888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.434952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12477.476241] Lustre: Skipped 1 previous similar message [12481.331431] Lustre: server umount lustre-OST0000 complete [12486.635162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12486.651893] LustreError: Skipped 1 previous similar message [12486.659539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.686323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.696692] Lustre: Skipped 1 previous similar message [12491.388101] Lustre: server umount lustre-MDT0000 complete [12492.774758] LustreError: 396426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12492.786542] LustreError: 396426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12495.349432] LustreError: 396433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648598 with bad export cookie 1146513010974059562 [12495.351177] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12495.370619] LustreError: 396433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12495.683775] Lustre: server umount lustre-MDT0001 complete [12505.042640] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12509.227721] Key type lgssc unregistered [12509.674911] LNet: 400107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12509.697449] LNetError: 400107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12509.732837] LNet: Removed LNI 192.168.204.109@tcp [12511.060228] Key type .llcrypt unregistered [12511.063811] Key type ._llcrypt unregistered [12529.385393] Key type ._llcrypt registered [12529.387892] Key type .llcrypt registered [12529.512632] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [12542.244871] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12543.042587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12543.061310] alg: No test for adler32 (adler32-zlib) [12544.086858] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [12544.294359] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12545.944837] Key type lgssc registered [12547.027553] Lustre: Echo OBD driver; http://www.lustre.org/ [12583.155334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12583.188198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.426678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12584.452754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12584.528628] Lustre: lustre-MDT0000: new disk, initializing [12584.653788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12584.677179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12589.664829] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12600.660493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.855742] Lustre: 403242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12600.981419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12600.984981] Lustre: Skipped 1 previous similar message [12601.048528] Lustre: lustre-MDT0001: new disk, initializing [12601.118132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.157772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12601.184863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12606.065130] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12615.590658] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12621.518793] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12630.832676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12631.215550] Lustre: lustre-OST0000: new disk, initializing [12631.220377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12631.225964] Lustre: 405210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12631.351515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12637.311697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12637.346197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12637.553548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12640.029595] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12650.538614] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12655.332505] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12655.550321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12660.117220] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12660.362215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12663.281192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12663.290412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12663.306666] Lustre: Skipped 1 previous similar message [12663.313434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12663.323272] Lustre: Skipped 1 previous similar message [12668.402022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12668.417722] Lustre: Skipped 1 previous similar message [12668.527996] Lustre: server umount lustre-OST0000 complete [12672.998268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12673.002692] LustreError: Skipped 1 previous similar message [12673.004973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12673.013078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12678.625165] LustreError: 403253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.641924] LustreError: 403253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12678.701636] Lustre: server umount lustre-MDT0000 complete [12683.099560] LustreError: 403234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648786 with bad export cookie 9866041758579036207 [12683.112392] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12683.430492] Lustre: server umount lustre-MDT0001 complete [12692.370290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12694.133861] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:53:14 (1788648794) [12700.705948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.249141] LustreError: 407671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12701.315424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12706.276280] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12706.786346] LustreError: 407672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.908704] LustreError: 407671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.870596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12716.535208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12722.512800] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12732.577923] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12740.207510] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12748.260047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12748.540355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12756.242726] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12763.982877] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12765.755950] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12772.349052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12777.441426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12777.453430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12777.468953] Lustre: Skipped 1 previous similar message [12777.485808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12777.498623] Lustre: Skipped 2 previous similar messages [12781.539324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12781.539770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12781.561259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.451720] Lustre: server umount lustre-OST0000 complete [12788.192342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12788.201153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.779542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12791.795822] Lustre: Skipped 3 previous similar messages [12793.263640] Lustre: server umount lustre-MDT0000 complete [12796.900773] LustreError: 407671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12796.916021] LustreError: 407671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12797.195081] LustreError: 407653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648900 with bad export cookie 9866041758579037089 [12797.206459] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12797.219243] LustreError: 407653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12797.601402] Lustre: server umount lustre-MDT0001 complete [12807.969467] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12812.160799] Key type lgssc unregistered [12812.592752] LNet: 411397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12812.600441] LNetError: 411397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12812.618780] LNet: Removed LNI 192.168.204.109@tcp [12813.546748] Key type .llcrypt unregistered [12813.548508] Key type ._llcrypt unregistered [12838.592888] Key type ._llcrypt registered [12838.601479] Key type .llcrypt registered [12838.717672] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12839.047443] Key type .llcrypt unregistered [12839.053287] Key type ._llcrypt unregistered [12858.290258] Key type ._llcrypt registered [12858.292938] Key type .llcrypt registered [12858.524568] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12859.484243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12859.623823] alg: No test for adler32 (adler32-zlib) [12860.774034] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [12861.125228] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12862.840139] Key type lgssc registered [12864.285960] Lustre: Echo OBD driver; http://www.lustre.org/ [12873.267042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12873.281991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12874.885605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12880.656857] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12884.449791] LustreError: 413601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12884.465090] LustreError: 413601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12888.776437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.016167] LustreError: 413598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12889.026749] LustreError: 413598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12889.043909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12894.178336] LustreError: 413602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12894.201479] LustreError: 413602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12895.313328] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12899.299046] LustreError: 413601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.309996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12899.317333] LustreError: 413601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12902.187952] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12904.430050] LustreError: 413602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.439044] LustreError: 413602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12908.410948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12908.756381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12913.283456] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12924.393773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12924.399706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12924.406498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12924.430289] Lustre: Skipped 1 previous similar message [12929.349499] Lustre: server umount lustre-OST0000 complete [12934.624640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12934.631957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12934.655558] Lustre: Skipped 1 previous similar message [12934.702433] Lustre: Skipped 2 previous similar messages [12939.746918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12939.769112] Lustre: Skipped 1 previous similar message [12940.202284] Lustre: server umount lustre-MDT0000 complete [12944.166050] LustreError: 413582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649047 with bad export cookie 904156062745610957 [12944.172657] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12944.174827] LustreError: 413582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12944.659986] Lustre: server umount lustre-MDT0001 complete [12956.327217] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12959.214947] Key type lgssc unregistered [12959.543640] LNet: 416749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12959.552073] LNetError: 416749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12959.575479] LNet: Removed LNI 192.168.204.109@tcp [12960.526598] Key type .llcrypt unregistered [12960.530799] Key type ._llcrypt unregistered [12983.708950] Key type ._llcrypt registered [12983.712715] Key type .llcrypt registered [12983.810651] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12984.738530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12984.844614] alg: No test for adler32 (adler32-zlib) [12985.949566] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [12986.158307] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12987.784802] Key type lgssc registered [12989.079873] Lustre: Echo OBD driver; http://www.lustre.org/ [12996.360492] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12996.374487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13011.872851] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13016.991821] LustreError: 418138:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.109@tcp: failed processing log, type 4: rc = -110 [13043.743358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13051.734469] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13061.158325] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13068.626769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.182140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13069.265039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13074.404309] LustreError: 418164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13074.423651] LustreError: 418164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13075.034650] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13078.520746] LustreError: 419177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.545473] LustreError: 419177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13083.623319] LustreError: 418163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13083.651715] LustreError: 418163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13085.252950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13085.768661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13090.893503] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13098.900590] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13104.671409] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13114.260089] Lustre: server umount lustre-OST0000 complete [13119.976819] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13119.991861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13120.008252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13121.507372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13121.509400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.625289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13126.637639] Lustre: Skipped 2 previous similar messages [13127.748348] Lustre: server umount lustre-MDT0000 complete [13131.586153] LustreError: 418144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649234 with bad export cookie 14812253157821720135 [13131.588201] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13131.591656] LustreError: 418144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13142.109374] Lustre: server umount lustre-MDT0001 complete [13153.130449] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13155.841312] Key type lgssc unregistered [13156.159080] LNet: 421708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13156.169687] LNetError: 421708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13156.187824] LNet: Removed LNI 192.168.204.109@tcp [13157.132949] Key type .llcrypt unregistered [13157.136079] Key type ._llcrypt unregistered [13186.543085] Key type ._llcrypt registered [13186.544848] Key type .llcrypt registered [13186.649840] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13187.580777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13187.719085] alg: No test for adler32 (adler32-zlib) [13188.901872] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [13189.185049] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13190.903252] Key type lgssc registered [13192.191335] Lustre: Echo OBD driver; http://www.lustre.org/ [13199.466333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13199.483724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13200.943925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13205.049896] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13210.081643] LustreError: 423135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.094084] LustreError: 423135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13212.206610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.501854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13216.516042] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13223.236921] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13228.636301] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13234.901598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13235.372473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13243.645322] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13245.629875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13251.105546] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13256.162403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.190078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13256.198164] Lustre: Skipped 2 previous similar messages [13259.083647] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [13261.287794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13261.304389] Lustre: Skipped 1 previous similar message [13262.483837] Lustre: server umount lustre-MDT0000 complete [13264.151253] LustreError: 423132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.926293] LustreError: 423117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.9@tcp arrived at 1788649367 with bad export cookie 16586921224277807661 [13264.946754] LustreError: 423117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13266.401584] LustreError: 423131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13266.412929] LustreError: 423131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13266.842733] LustreError: 423116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649369 with bad export cookie 16586921224277806765 [13266.856772] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13268.455278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13268.466595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13268.486355] Lustre: Skipped 2 previous similar messages [13268.491448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13271.524640] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13271.552195] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13273.112869] Lustre: server umount lustre-OST0000 complete [13276.648415] LustreError: 423132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13276.674532] LustreError: 423132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13281.094312] Lustre: server umount lustre-MDT0001 complete [13292.760583] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13295.628415] Key type lgssc unregistered [13295.962542] LNet: 426818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13295.977513] LNetError: 426818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13295.997099] LNet: Removed LNI 192.168.204.109@tcp [13296.882991] Key type .llcrypt unregistered [13296.887255] Key type ._llcrypt unregistered [13323.338248] Key type ._llcrypt registered [13323.342720] Key type .llcrypt registered [13323.513699] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13324.897609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13324.969886] alg: No test for adler32 (adler32-zlib) [13326.141623] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [13326.440881] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13328.207173] Key type lgssc registered [13329.293861] Lustre: Echo OBD driver; http://www.lustre.org/ [13338.691699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13338.746447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13340.465230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13347.114413] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13350.885823] LustreError: 428231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13350.911771] LustreError: 428231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13356.034030] LustreError: 428232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13357.029120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13357.670941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13362.775876] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13371.211527] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13377.504992] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13385.442943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13385.941327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13393.246072] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13402.536129] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13410.487794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13418.980154] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13418.984664] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13418.998507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13419.838564] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [13421.026270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13421.028619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.912872] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [13424.923916] Lustre: Skipped 1 previous similar message [13430.042348] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [13430.050119] Lustre: Skipped 2 previous similar messages [13432.289969] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13432.447060] Lustre: server umount lustre-MDT0001 complete [13436.385015] LustreError: 428444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13436.408650] LustreError: 428444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13441.426495] Lustre: server umount lustre-OST0000 complete [13441.508510] LustreError: 428232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13455.673447] Lustre: server umount lustre-MDT0000 complete [13471.154885] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13474.284874] Key type lgssc unregistered [13474.709956] LNet: 431941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13474.716740] LNetError: 431941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13475.758740] LNet: Removed LNI 192.168.204.109@tcp [13476.842576] Key type .llcrypt unregistered [13476.852141] Key type ._llcrypt unregistered [13501.804209] Key type ._llcrypt registered [13501.805810] Key type .llcrypt registered [13501.879436] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [13516.074393] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13516.978500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13517.017448] alg: No test for adler32 (adler32-zlib) [13518.108379] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [13518.364597] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13520.048443] Key type lgssc registered [13521.287053] Lustre: Echo OBD driver; http://www.lustre.org/ [13564.692130] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13583.232506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13583.261738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.604905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13584.669060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13584.899282] Lustre: lustre-MDT0000: new disk, initializing [13585.101887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13585.128258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13591.264517] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13609.546073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13609.875619] Lustre: lustre-OST0000: new disk, initializing [13609.881596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13609.885087] Lustre: Skipped 1 previous similar message [13609.893567] Lustre: 436973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13609.992390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13617.097828] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13617.782121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13617.796965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13617.829079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13626.977365] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13645.780164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13645.947300] Lustre: 435956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13646.025771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13646.107301] Lustre: lustre-MDT0001: new disk, initializing [13646.238698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13646.289557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13646.301729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13652.379682] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13664.857812] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13665.233925] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13669.965240] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13670.222561] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13677.026318] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13677.044976] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13677.050961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13677.060286] Lustre: Skipped 1 previous similar message [13682.149698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13682.155802] Lustre: Skipped 3 previous similar messages [13687.266606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13687.293643] Lustre: Skipped 2 previous similar messages [13689.312087] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13689.586443] Lustre: server umount lustre-MDT0001 complete [13697.507663] LustreError: 435966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.535698] LustreError: 435966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13702.631351] LustreError: 435962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13702.649048] LustreError: 435962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13707.756583] LustreError: 437353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13707.799082] LustreError: 437353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13709.793107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13709.801162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13709.813715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13709.817548] Lustre: Skipped 1 previous similar message [13712.868351] LustreError: 435967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13715.536351] Lustre: server umount lustre-OST0000 complete [13717.985752] LustreError: 435966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13734.367417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13734.912620] Lustre: server umount lustre-MDT0000 complete [13751.145209] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13754.791856] Key type lgssc unregistered [13755.146268] LNet: 440721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13755.153555] LNetError: 440721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13755.177311] LNet: Removed LNI 192.168.204.109@tcp [13756.589127] Key type .llcrypt unregistered [13756.593413] Key type ._llcrypt unregistered [13816.468808] Key type ._llcrypt registered [13816.477102] Key type .llcrypt registered [13816.668793] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13817.281632] Key type .llcrypt unregistered [13817.283293] Key type ._llcrypt unregistered [13844.947876] Key type ._llcrypt registered [13844.949706] Key type .llcrypt registered [13845.227811] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13847.355916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13847.499905] alg: No test for adler32 (adler32-zlib) [13849.298808] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [13849.617171] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13851.271253] Key type lgssc registered [13852.712054] Lustre: Echo OBD driver; http://www.lustre.org/ [13885.258773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13885.274913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13886.453588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13886.475260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13886.593787] Lustre: lustre-MDT0000: new disk, initializing [13886.723167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13886.741690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13891.242398] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13903.380565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13903.578841] Lustre: 444098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13903.685924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13903.696736] Lustre: Skipped 1 previous similar message [13903.845402] Lustre: lustre-MDT0001: new disk, initializing [13903.989138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13904.044028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13904.055570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13908.917692] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13918.101759] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13924.424348] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13933.884275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13934.277526] Lustre: lustre-OST0000: new disk, initializing [13934.292900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13934.314851] Lustre: 446068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13934.520685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13940.847948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13940.867607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13941.052764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13941.367720] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13954.338901] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13965.294634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13965.313900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13965.364472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13966.826475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13966.830790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13969.905391] Lustre: server umount lustre-MDT0000 complete [13973.815599] LustreError: 444090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650076 with bad export cookie 8638745696908000544 [13973.818458] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13973.833458] LustreError: 444090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13974.563291] Lustre: server umount lustre-MDT0001 complete [13989.247073] Lustre: server umount lustre-OST0000 complete [14002.786901] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:15:02 (1788650102) [14013.156566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14013.800671] LustreError: 448096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14013.839734] LustreError: 448096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14013.957331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14019.042890] LustreError: 448097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.493742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14023.142220] LustreError: 448096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14028.257049] LustreError: 448097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14032.199212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14032.833406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14038.272797] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14047.592990] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14054.846842] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14076.082849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14076.418748] Lustre: Found index 0 for lustre-OST0000, updating log [14076.732584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14083.891944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14084.842608] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14093.224242] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14098.400142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14098.405647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14098.419245] Lustre: Skipped 1 previous similar message [14098.431474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14098.434023] Lustre: Skipped 2 previous similar messages [14104.103657] Lustre: server umount lustre-OST0000 complete [14120.935323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14120.946575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14120.970946] Lustre: Skipped 2 previous similar messages [14120.984646] Lustre: Skipped 4 previous similar messages [14125.091327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14125.097314] Lustre: Skipped 1 previous similar message [14127.219419] Lustre: server umount lustre-MDT0000 complete [14131.173759] LustreError: 448825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14131.195864] LustreError: 448825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14132.009823] LustreError: 448077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650234 with bad export cookie 8638745696908001944 [14132.017430] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14132.021172] LustreError: 448077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14132.444907] Lustre: server umount lustre-MDT0001 complete [14137.439251] LNet: 1 local NIs in recovery (showing 1): 192.168.204.109@tcp [14144.457716] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:17:24 (1788650244) [14772.079847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14772.594686] LustreError: 454509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14772.690761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14777.376173] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14777.825755] LustreError: 454510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14782.953394] LustreError: 454509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14787.923540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14788.590026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14795.022763] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14804.188935] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14812.328785] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14829.663156] LNet: 1 local NIs in recovery (showing 1): 192.168.204.109@tcp [14833.579175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14833.910314] Lustre: Found index 0 for lustre-OST0000, updating log [14834.285771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14841.239144] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14848.820861] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14852.066071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14852.075106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14852.094283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14852.100193] Lustre: Skipped 2 previous similar messages [14857.441536] Lustre: server umount lustre-OST0000 complete [14862.122913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14862.138174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14862.150698] Lustre: Skipped 1 previous similar message [14864.225784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14864.237597] Lustre: Skipped 4 previous similar messages [14867.304187] LustreError: 454509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.320194] LustreError: 454509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14867.523529] Lustre: server umount lustre-MDT0000 complete [14871.342747] LustreError: 454491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650974 with bad export cookie 8638745696908003029 [14871.354858] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14871.659142] Lustre: server umount lustre-MDT0001 complete [14889.057449] LNet: 1 local NIs in recovery (showing 1): 192.168.204.109@tcp [14904.904708] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [14915.921277] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [14956.986925] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:30:57 (1788651057) [14973.756680] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14977.014637] Key type lgssc unregistered [14977.358323] LNet: 461124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14977.380526] LNetError: 461124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15095.147493] LNet: Removed LNI 192.168.204.109@tcp [15096.059118] Key type .llcrypt unregistered [15096.063042] Key type ._llcrypt unregistered [15111.554978] Key type ._llcrypt registered [15111.557233] Key type .llcrypt registered [15111.639481] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [15112.438816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15112.474822] alg: No test for adler32 (adler32-zlib) [15113.541760] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [15113.748036] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [15115.463214] Key type lgssc registered [15116.623808] Lustre: Echo OBD driver; http://www.lustre.org/ [15120.584776] LNet: Added LNI 192.168.204.109@tcp100 [8/256/0/180] [15121.264598] LNet: Added LNI 192.168.204.109@tcp101 [8/256/0/180] [15122.140273] LNet: Added LNI 192.168.204.109@tcp102 [8/256/0/180] [15122.818261] LNet: Added LNI 192.168.204.109@tcp103 [8/256/0/180] [15123.606250] LNet: Added LNI 192.168.204.109@tcp104 [8/256/0/180] [15124.445893] LNet: Added LNI 192.168.204.109@tcp105 [8/256/0/180] [15125.240447] LNet: Added LNI 192.168.204.109@tcp106 [8/256/0/180] [15126.029597] LNet: Added LNI 192.168.204.109@tcp107 [8/256/0/180] [15126.836797] LNet: Added LNI 192.168.204.109@tcp108 [8/256/0/180] [15127.657961] LNet: Added LNI 192.168.204.109@tcp109 [8/256/0/180] [15128.524575] LNet: Added LNI 192.168.204.109@tcp110 [8/256/0/180] [15129.242937] LNet: Added LNI 192.168.204.109@tcp111 [8/256/0/180] [15129.972928] LNet: Added LNI 192.168.204.109@tcp112 [8/256/0/180] [15130.718105] LNet: Added LNI 192.168.204.109@tcp113 [8/256/0/180] [15131.607987] LNet: Added LNI 192.168.204.109@tcp114 [8/256/0/180] [15132.587923] LNet: Added LNI 192.168.204.109@tcp115 [8/256/0/180] [15133.493420] LNet: Added LNI 192.168.204.109@tcp116 [8/256/0/180] [15134.453109] LNet: Added LNI 192.168.204.109@tcp117 [8/256/0/180] [15135.347324] LNet: Added LNI 192.168.204.109@tcp118 [8/256/0/180] [15136.170154] LNet: Added LNI 192.168.204.109@tcp119 [8/256/0/180] [15136.927486] LNet: Added LNI 192.168.204.109@tcp120 [8/256/0/180] [15137.766226] LNet: Added LNI 192.168.204.109@tcp121 [8/256/0/180] [15138.701240] LNet: Added LNI 192.168.204.109@tcp122 [8/256/0/180] [15139.506927] LNet: Added LNI 192.168.204.109@tcp123 [8/256/0/180] [15140.475520] LNet: Added LNI 192.168.204.109@tcp124 [8/256/0/180] [15141.439769] LNet: Added LNI 192.168.204.109@tcp125 [8/256/0/180] [15142.504911] LNet: Added LNI 192.168.204.109@tcp126 [8/256/0/180] [15143.432927] LNet: Added LNI 192.168.204.109@tcp127 [8/256/0/180] [15144.379492] LNet: Added LNI 192.168.204.109@tcp128 [8/256/0/180] [15145.336317] LNet: Added LNI 192.168.204.109@tcp129 [8/256/0/180] [15146.851627] LNet: Added LNI 192.168.204.109@tcp130 [8/256/0/180] [15148.187698] LNet: Added LNI 192.168.204.109@tcp131 [8/256/0/180] [15149.179835] LNet: Added LNI 192.168.204.109@tcp132 [8/256/0/180] [15150.137462] LNet: Added LNI 192.168.204.109@tcp133 [8/256/0/180] [15151.079956] LNet: Added LNI 192.168.204.109@tcp134 [8/256/0/180] [15151.850348] LNet: Added LNI 192.168.204.109@tcp135 [8/256/0/180] [15158.298272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15158.346776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15159.476272] Lustre: 464152:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.109@tcp: can't add NID 192.168.204.109@tcp130: rc = -75 [15159.543528] LNetError: 464176:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15159.550766] LNetError: Protocol error connecting to 192.168.204.109@tcp120 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15159.562101] LNetError: Skipped 6 previous similar messages [15159.632623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15159.663757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15159.769986] Lustre: lustre-MDT0000: new disk, initializing [15159.842903] LNet: 9 local NIs in recovery (showing 9): 192.168.204.109@tcp122, 192.168.204.109@tcp124, 192.168.204.109@tcp125, 192.168.204.109@tcp126, 192.168.204.109@tcp127, 192.168.204.109@tcp128, 192.168.204.109@tcp129, 192.168.204.109@tcp120, 192.168.204.109@tcp130 [15159.948075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15159.991979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15160.609903] LNetError: 461608:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15160.635684] LNetError: 461608:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15160.650477] LNetError: Protocol error connecting to 192.168.204.109@tcp111 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15160.664985] LNetError: Skipped 6 previous similar messages [15162.979558] LNetError: 464176:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15163.000756] LNetError: 464176:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15163.029967] LNetError: Protocol error connecting to 192.168.204.109@tcp124 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15163.052057] LNetError: Skipped 14 previous similar messages [15165.086748] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15174.246788] LNetError: 464349:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15174.275866] LNetError: 464349:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15174.292476] LNetError: Protocol error connecting to 192.168.204.109@tcp132 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15174.327205] LNetError: Skipped 3 previous similar messages [15178.049127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15178.243719] Lustre: 464167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15178.323621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15178.330451] Lustre: Skipped 1 previous similar message [15178.426366] Lustre: lustre-MDT0001: new disk, initializing [15178.459671] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15178.486042] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15178.487512] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.109@tcp: can't add NID 192.168.204.109@tcp130: rc = -75 [15178.495934] LNetError: Protocol error connecting to 192.168.204.109@tcp134 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15178.502267] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15178.700747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15178.771030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15178.779741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15183.984941] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15187.042652] Lustre: 465349:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.109@tcp: can't add NID 192.168.204.109@tcp130: rc = -75 [15187.055281] Lustre: 465349:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15190.573507] LNetError: 461610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15190.592869] LNetError: 461610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15190.601377] LNetError: Protocol error connecting to 192.168.204.109@tcp123 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15190.614595] LNetError: Skipped 7 previous similar messages [15194.135847] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15199.525146] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15206.213936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15206.533714] Lustre: lustre-OST0000: new disk, initializing [15206.543137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15206.556876] Lustre: 466146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15206.672600] Lustre: 466151:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.109@tcp: can't add NID 192.168.204.109@tcp130: rc = -75 [15206.686814] Lustre: 466151:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15206.757654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15206.950079] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15206.959957] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15206.970204] LNetError: Protocol error connecting to 192.168.204.109@tcp121 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15206.978823] LNetError: Skipped 13 previous similar messages [15212.961198] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15214.619049] Lustre: 466581:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.109@tcp: can't add NID 192.168.204.109@tcp130: rc = -75 [15214.633106] Lustre: 466581:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15214.740192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15214.763107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15214.943870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15219.295222] LNet: 4 local NIs in recovery (showing 4): 192.168.204.109@tcp134, 192.168.204.109@tcp135, 192.168.204.109@tcp124, 192.168.204.109@tcp132 [15223.470462] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15235.553942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15235.567322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15235.575108] LustreError: Skipped 1 previous similar message [15235.577493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15235.613052] Lustre: Skipped 1 previous similar message [15237.483563] Lustre: server umount lustre-OST0000 complete [15240.738356] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.109 [15240.759123] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [15240.773797] LNetError: Protocol error connecting to 192.168.204.109@tcp120 at host 192.168.204.109:988: is it running a compatible version of Lustre? [15240.784475] LNetError: Skipped 19 previous similar messages [15245.287625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15245.294260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15245.307185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15245.312392] Lustre: Skipped 1 previous similar message [15247.990453] Lustre: server umount lustre-MDT0000 complete [15252.227654] LustreError: 464162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651355 with bad export cookie 7714841219673185808 [15252.237327] LustreError: 464162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15252.237923] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15252.791713] Lustre: server umount lustre-MDT0001 complete [15266.631718] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [15270.908858] Key type lgssc unregistered [15271.495223] LNet: 468056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15271.508053] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15271.522120] LNet: Removed LNI 192.168.204.109@tcp [15272.544358] LNet: Removed LNI 192.168.204.109@tcp100 [15272.547615] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15272.554533] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15272.562620] LNet: Removed LNI 192.168.204.109@tcp101 [15273.634341] LNet: Removed LNI 192.168.204.109@tcp102 [15273.637536] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15273.663749] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15273.682039] LNet: Removed LNI 192.168.204.109@tcp103 [15274.719527] LNet: Removed LNI 192.168.204.109@tcp104 [15275.744212] LNet: Removed LNI 192.168.204.109@tcp105 [15275.751886] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15275.759879] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15275.767356] LNet: Removed LNI 192.168.204.109@tcp106 [15276.831266] LNet: Removed LNI 192.168.204.109@tcp107 [15277.855490] LNet: Removed LNI 192.168.204.109@tcp108 [15278.879190] LNet: Removed LNI 192.168.204.109@tcp109 [15279.903360] LNet: Removed LNI 192.168.204.109@tcp110 [15279.910110] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15279.917610] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15279.936396] LNet: Removed LNI 192.168.204.109@tcp111 [15280.991196] LNet: Removed LNI 192.168.204.109@tcp112 [15282.017264] LNet: Removed LNI 192.168.204.109@tcp113 [15283.039762] LNet: Removed LNI 192.168.204.109@tcp114 [15284.063184] LNet: Removed LNI 192.168.204.109@tcp115 [15285.087783] LNet: Removed LNI 192.168.204.109@tcp116 [15286.111175] LNet: Removed LNI 192.168.204.109@tcp117 [15287.135832] LNet: Removed LNI 192.168.204.109@tcp118 [15288.160795] LNet: Removed LNI 192.168.204.109@tcp119 [15288.163636] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15288.171030] LNetError: 468056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15288.187445] LNet: Removed LNI 192.168.204.109@tcp120 [15289.247179] LNet: Removed LNI 192.168.204.109@tcp121 [15290.271465] LNet: Removed LNI 192.168.204.109@tcp122 [15291.295211] LNet: Removed LNI 192.168.204.109@tcp123 [15291.299199] LNet: Removed LNI 192.168.204.109@tcp124 [15292.319133] LNet: Removed LNI 192.168.204.109@tcp125 [15293.343609] LNet: Removed LNI 192.168.204.109@tcp126 [15294.370101] LNet: Removed LNI 192.168.204.109@tcp127 [15295.391721] LNet: Removed LNI 192.168.204.109@tcp128 [15296.415353] LNet: Removed LNI 192.168.204.109@tcp129 [15297.439264] LNet: Removed LNI 192.168.204.109@tcp130 [15298.463364] LNet: Removed LNI 192.168.204.109@tcp131 [15298.467103] LNet: Removed LNI 192.168.204.109@tcp132 [15299.487240] LNet: Removed LNI 192.168.204.109@tcp133 [15299.497680] LNet: Removed LNI 192.168.204.109@tcp134 [15299.525626] LNet: Removed LNI 192.168.204.109@tcp135 [15300.506596] Key type .llcrypt unregistered [15300.514644] Key type ._llcrypt unregistered [15325.047416] Key type ._llcrypt registered [15325.051597] Key type .llcrypt registered [15325.205232] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [15339.091389] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [15340.565947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15340.617224] alg: No test for adler32 (adler32-zlib) [15341.680465] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [15341.903251] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [15343.567179] Key type lgssc registered [15344.757952] Lustre: Echo OBD driver; http://www.lustre.org/ [15381.030441] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:38:01 (1788651481) [15401.879266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [15405.862664] Key type lgssc unregistered [15406.240678] LNet: 472134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15406.255016] LNetError: 472134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15406.293046] LNet: Removed LNI 192.168.204.109@tcp [15407.346970] Key type .llcrypt unregistered [15407.348418] Key type ._llcrypt unregistered [15426.527042] Key type ._llcrypt registered [15426.529526] Key type .llcrypt registered [15426.650841] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [15427.947053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15427.988164] alg: No test for adler32 (adler32-zlib) [15429.072905] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [15429.368678] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [15431.087241] Key type lgssc registered [15432.354532] Lustre: Echo OBD driver; http://www.lustre.org/ [15443.208794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15443.235106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15444.521800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15444.588839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15444.746614] Lustre: lustre-MDT0000: new disk, initializing [15444.873918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15444.911018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15450.926786] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15464.862666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15465.074606] Lustre: 473346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15465.152520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15465.166198] Lustre: Skipped 1 previous similar message [15465.311829] Lustre: lustre-MDT0001: new disk, initializing [15465.419992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15465.470267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15465.492281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15471.837719] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15484.521491] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15491.462971] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15499.820404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15500.113903] Lustre: lustre-OST0000: new disk, initializing [15500.118733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15500.130800] Lustre: 475316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15500.239431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15507.111786] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15510.196343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15510.218774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15510.344745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15518.190783] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15523.966176] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing lnet_if_list [15531.827517] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15531.875634] LNet: Added LNI 192.168.204.109@tcp10 [8/256/0/180] [15539.806835] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15539.858978] LNet: Added LNI 192.168.204.109@tcp11 [8/256/0/180] [15544.682594] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15544.703521] LNet: Added LNI 192.168.204.109@tcp12 [8/256/0/180] [15549.326390] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15549.364803] LNet: Added LNI 192.168.204.109@tcp13 [8/256/0/180] [15554.885035] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15554.924726] LNet: Added LNI 192.168.204.109@tcp14 [8/256/0/180] [15560.335231] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15560.372830] LNet: Added LNI 192.168.204.109@tcp15 [8/256/0/180] [15565.751635] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15565.779297] LNet: Added LNI 192.168.204.109@tcp16 [8/256/0/180] [15570.617813] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15570.644459] LNet: Added LNI 192.168.204.109@tcp17 [8/256/0/180] [15575.895204] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15575.922557] LNet: Added LNI 192.168.204.109@tcp18 [8/256/0/180] [15581.417482] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15581.450751] LNet: Added LNI 192.168.204.109@tcp19 [8/256/0/180] [15587.390158] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15587.426423] LNet: Added LNI 192.168.204.109@tcp20 [8/256/0/180] [15595.062394] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15595.093351] LNet: Added LNI 192.168.204.109@tcp21 [8/256/0/180] [15603.052137] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15603.080132] LNet: Added LNI 192.168.204.109@tcp22 [8/256/0/180] [15609.697550] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15609.727883] LNet: Added LNI 192.168.204.109@tcp23 [8/256/0/180] [15616.178322] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15616.205712] LNet: Added LNI 192.168.204.109@tcp24 [8/256/0/180] [15622.620992] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15622.655713] LNet: Added LNI 192.168.204.109@tcp25 [8/256/0/180] [15629.528097] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15629.552356] LNet: Added LNI 192.168.204.109@tcp26 [8/256/0/180] [15637.042984] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15637.092873] LNet: Added LNI 192.168.204.109@tcp27 [8/256/0/180] [15644.053148] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15644.098311] LNet: Added LNI 192.168.204.109@tcp28 [8/256/0/180] [15652.780572] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15652.862852] LNet: Added LNI 192.168.204.109@tcp29 [8/256/0/180] [15660.273225] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15660.297672] LNet: Added LNI 192.168.204.109@tcp30 [8/256/0/180] [15667.219932] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15667.248432] LNet: Added LNI 192.168.204.109@tcp31 [8/256/0/180] [15674.300763] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15674.357876] LNet: Added LNI 192.168.204.109@tcp32 [8/256/0/180] [15681.524415] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15681.551173] LNet: Added LNI 192.168.204.109@tcp33 [8/256/0/180] [15689.090549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15689.121122] LNet: Added LNI 192.168.204.109@tcp34 [8/256/0/180] [15695.908344] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15695.935313] LNet: Added LNI 192.168.204.109@tcp35 [8/256/0/180] [15703.222690] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15703.263257] LNet: Added LNI 192.168.204.109@tcp36 [8/256/0/180] [15710.950772] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15710.987698] LNet: Added LNI 192.168.204.109@tcp37 [8/256/0/180] [15718.821524] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15718.851719] LNet: Added LNI 192.168.204.109@tcp38 [8/256/0/180] [15726.742363] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15726.790450] LNet: Added LNI 192.168.204.109@tcp39 [8/256/0/180] [15734.355883] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15734.408867] LNet: Added LNI 192.168.204.109@tcp40 [8/256/0/180] [15743.337732] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15743.383158] LNet: Added LNI 192.168.204.109@tcp41 [8/256/0/180] [15752.550365] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15752.608697] LNet: Added LNI 192.168.204.109@tcp42 [8/256/0/180] [15761.598844] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15761.652525] LNet: Added LNI 192.168.204.109@tcp43 [8/256/0/180] [15771.285948] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15771.330824] LNet: Added LNI 192.168.204.109@tcp44 [8/256/0/180] [15779.613631] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15779.659281] LNet: Added LNI 192.168.204.109@tcp45 [8/256/0/180] [15788.251655] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15788.290611] LNet: Added LNI 192.168.204.109@tcp46 [8/256/0/180] [15797.003346] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15797.042991] LNet: Added LNI 192.168.204.109@tcp47 [8/256/0/180] [15806.345426] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15806.380476] LNet: Added LNI 192.168.204.109@tcp48 [8/256/0/180] [15815.785697] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15815.816546] LNet: Added LNI 192.168.204.109@tcp49 [8/256/0/180] [15827.155664] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15827.212606] LNet: Added LNI 192.168.204.109@tcp50 [8/256/0/180] [15848.417314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15848.430078] Lustre: Skipped 1 previous similar message [15848.433134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15848.441175] Lustre: Skipped 2 previous similar messages [15853.553047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15853.576931] Lustre: Skipped 2 previous similar messages [15853.945658] Lustre: server umount lustre-MDT0000 complete [15857.926552] LustreError: 473337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651960 with bad export cookie 2616139369180726321 [15857.930791] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15857.954503] LustreError: 473337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15858.255719] Lustre: server umount lustre-MDT0001 complete [15874.975602] Lustre: 472614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651961/real 1788651961] req@00000000a754773b x1875537068998272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15875.009244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15875.019420] Lustre: Skipped 1 previous similar message [15878.111670] Lustre: 472615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651965/real 1788651965] req@000000003f4dcbf0 x1875537068998400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15878.477550] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [15879.144694] Lustre: 472616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651966/real 1788651966] req@0000000084b7b0a0 x1875537068998528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15883.170184] Lustre: 472614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651970/real 1788651970] req@0000000008fb70a8 x1875537068998784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15888.360156] Lustre: 472613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651975/real 1788651975] req@0000000048a789c8 x1875537068998912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15888.919685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15903.149742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15904.162100] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15908.724301] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15909.348987] LustreError: 475310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15909.370418] LustreError: 475310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15914.469549] LustreError: 475309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15919.410642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15919.587329] LustreError: 499562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15919.633487] LustreError: 499562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15919.945398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15924.883295] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [15925.227406] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15932.521073] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15939.414559] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15959.522726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15959.534539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15959.561468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15961.076451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15966.177514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15966.200912] Lustre: Skipped 3 previous similar messages [15971.298583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15973.348573] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15973.637730] Lustre: server umount lustre-OST0000 complete [15981.536287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15981.544541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15981.558222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15981.558231] Lustre: Skipped 1 previous similar message [15981.583383] Lustre: Skipped 2 previous similar messages [15984.529708] Lustre: server umount lustre-MDT0000 complete [15986.656572] LustreError: 499566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15986.675314] LustreError: 499566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15988.428442] LustreError: 474150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652091 with bad export cookie 2616139369180728953 [15988.440331] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15988.456991] LustreError: 474150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15988.975594] Lustre: server umount lustre-MDT0001 complete [16000.158711] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [16003.107531] Key type lgssc unregistered [16003.560969] LNet: 502363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16003.573799] LNetError: 502363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16003.581353] LNet: Removed LNI 192.168.204.109@tcp [16003.589452] LNet: Removed LNI 192.168.204.109@tcp10 [16003.592251] LNet: Removed LNI 192.168.204.109@tcp11 [16003.596670] LNet: Removed LNI 192.168.204.109@tcp12 [16003.600212] LNet: Removed LNI 192.168.204.109@tcp13 [16003.604483] LNet: Removed LNI 192.168.204.109@tcp14 [16003.608151] LNet: Removed LNI 192.168.204.109@tcp15 [16003.611366] LNet: Removed LNI 192.168.204.109@tcp16 [16003.613782] LNet: Removed LNI 192.168.204.109@tcp17 [16003.617634] LNet: Removed LNI 192.168.204.109@tcp18 [16003.621810] LNet: Removed LNI 192.168.204.109@tcp19 [16003.625867] LNet: Removed LNI 192.168.204.109@tcp20 [16003.633062] LNet: Removed LNI 192.168.204.109@tcp21 [16003.638145] LNet: Removed LNI 192.168.204.109@tcp22 [16003.642220] LNet: Removed LNI 192.168.204.109@tcp23 [16003.644843] LNet: Removed LNI 192.168.204.109@tcp24 [16003.647873] LNet: Removed LNI 192.168.204.109@tcp25 [16003.652242] LNet: Removed LNI 192.168.204.109@tcp26 [16003.655522] LNet: Removed LNI 192.168.204.109@tcp27 [16003.658748] LNet: Removed LNI 192.168.204.109@tcp28 [16003.661736] LNet: Removed LNI 192.168.204.109@tcp29 [16003.665212] LNet: Removed LNI 192.168.204.109@tcp30 [16003.669704] LNet: Removed LNI 192.168.204.109@tcp31 [16003.672290] LNet: Removed LNI 192.168.204.109@tcp32 [16003.675513] LNet: Removed LNI 192.168.204.109@tcp33 [16003.679197] LNet: Removed LNI 192.168.204.109@tcp34 [16003.685308] LNet: Removed LNI 192.168.204.109@tcp35 [16003.689514] LNet: Removed LNI 192.168.204.109@tcp36 [16003.693833] LNet: Removed LNI 192.168.204.109@tcp37 [16003.699972] LNet: Removed LNI 192.168.204.109@tcp38 [16003.702806] LNet: Removed LNI 192.168.204.109@tcp39 [16003.707215] LNet: Removed LNI 192.168.204.109@tcp40 [16003.711626] LNet: Removed LNI 192.168.204.109@tcp41 [16003.717292] LNet: Removed LNI 192.168.204.109@tcp42 [16003.727235] LNet: Removed LNI 192.168.204.109@tcp43 [16003.730974] LNet: Removed LNI 192.168.204.109@tcp44 [16003.735759] LNet: Removed LNI 192.168.204.109@tcp45 [16003.740334] LNet: Removed LNI 192.168.204.109@tcp46 [16003.743980] LNet: Removed LNI 192.168.204.109@tcp47 [16003.749210] LNet: Removed LNI 192.168.204.109@tcp48 [16003.751977] LNet: Removed LNI 192.168.204.109@tcp49 [16003.768687] LNet: Removed LNI 192.168.204.109@tcp50 [16004.572174] Key type .llcrypt unregistered [16004.577190] Key type ._llcrypt unregistered [16025.645933] Key type ._llcrypt registered [16025.649096] Key type .llcrypt registered [16025.770519] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [16040.969281] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [16042.316259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16042.349556] alg: No test for adler32 (adler32-zlib) [16043.522331] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [16043.807059] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [16045.551211] Key type lgssc registered [16046.940940] Lustre: Echo OBD driver; http://www.lustre.org/ [16090.444399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16090.464439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16091.791959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16091.844992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16091.959305] Lustre: lustre-MDT0000: new disk, initializing [16092.081072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16092.105520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16096.963517] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16110.186570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16110.374580] Lustre: 505558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16110.450817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16110.458552] Lustre: Skipped 1 previous similar message [16110.579806] Lustre: lustre-MDT0001: new disk, initializing [16110.770726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16110.829946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16110.860861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16116.059424] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16125.297495] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16131.765511] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16139.408135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16139.604662] Lustre: *** cfs_fail_loc=90f, val=0*** [16139.675408] Lustre: lustre-OST0000: new disk, initializing [16139.680668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16139.692577] Lustre: 507573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16139.806123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16145.491889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16145.500646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16145.619222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16146.287866] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16155.228929] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16166.374935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16166.380786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16166.398117] Lustre: Skipped 1 previous similar message [16166.402040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16168.553746] Lustre: server umount lustre-OST0000 complete [16176.612577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16176.613997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16176.633737] Lustre: Skipped 1 previous similar message [16176.642712] Lustre: Skipped 2 previous similar messages [16178.553300] Lustre: server umount lustre-MDT0000 complete [16182.249254] LustreError: 505548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652285 with bad export cookie 16952493789806212997 [16182.252888] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16182.261243] LustreError: 505548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16182.546689] Lustre: server umount lustre-MDT0001 complete [16192.577214] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [16195.567591] Key type lgssc unregistered [16195.844144] LNet: 509477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16195.857488] LNetError: 509477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16195.885181] LNet: Removed LNI 192.168.204.109@tcp [16196.732433] Key type .llcrypt unregistered [16196.734860] Key type ._llcrypt unregistered [16215.824935] Key type ._llcrypt registered [16215.827524] Key type .llcrypt registered [16215.917327] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [16216.830652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16216.849065] alg: No test for adler32 (adler32-zlib) [16217.896525] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [16218.193927] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [16219.855350] Key type lgssc registered [16220.823994] Lustre: Echo OBD driver; http://www.lustre.org/ [16228.077688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16228.092312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16229.684065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16235.407864] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16239.074256] LustreError: 510750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16239.107298] LustreError: 510750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16244.196329] LustreError: 510751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16246.760989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16247.310375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16252.384476] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16260.689517] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16266.908325] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16273.421732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16273.756468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16279.560744] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16287.163122] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16303.563355] loop0: detected capacity change from 0 to 204800000 [16303.779396] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16303.792270] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16303.827369] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.806438] loop0: detected capacity change from 0 to 204800000 [16310.985612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16311.000958] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16311.013858] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16316.191706] loop0: detected capacity change from 0 to 204800000 [16321.983093] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16322.169843] Lustre: fs15246-OST0000: new disk, initializing [16322.178695] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16322.192977] Lustre: 514343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16322.254196] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16328.626227] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16339.834962] loop1: detected capacity change from 0 to 204800000 [16346.430208] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16346.526284] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16346.604244] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16346.682923] Lustre: fs15246-MDT0000: new disk, initializing [16346.829491] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16346.878644] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16347.044637] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16348.069705] Lustre: 515467:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000cef721e x1875537896195072/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788652467 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16351.885552] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16353.157210] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16353.183657] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16353.233780] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16377.352840] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16377.378489] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16379.935491] Lustre: server umount fs15246-MDT0000 complete [16382.442669] LustreError: 515889:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16387.557819] LustreError: 511734:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16390.593263] Lustre: server umount fs15246-OST0000 complete [16404.967875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16404.976068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16404.986811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16408.034975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16408.039253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16408.047037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16408.067672] Lustre: Skipped 1 previous similar message [16413.159842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16413.169413] Lustre: Skipped 1 previous similar message [16418.275718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16418.281229] Lustre: Skipped 1 previous similar message [16419.298314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16419.445693] Lustre: server umount lustre-OST0000 complete [16426.465776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16426.480648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16426.500579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16429.973455] Lustre: server umount lustre-MDT0000 complete [16433.633093] LustreError: 510744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16433.654853] LustreError: 510744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16434.490093] LustreError: 510731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652537 with bad export cookie 5212141312202191535 [16434.501158] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16434.503948] LustreError: 510731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16434.821311] Lustre: server umount lustre-MDT0001 complete [16447.219878] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [16450.901867] Key type lgssc unregistered [16451.227059] LNet: 518565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16451.239241] LNetError: 518565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16451.257085] LNet: Removed LNI 192.168.204.109@tcp [16452.298246] Key type .llcrypt unregistered [16452.302269] Key type ._llcrypt unregistered [16471.481940] Key type ._llcrypt registered [16471.484574] Key type .llcrypt registered [16471.619113] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [16472.542149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16472.613594] alg: No test for adler32 (adler32-zlib) [16473.941486] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [16474.295198] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [16475.991179] Key type lgssc registered [16477.888276] Lustre: Echo OBD driver; http://www.lustre.org/ [16524.859581] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [16537.522747] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [16584.071970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16584.132905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16585.423252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16585.445579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16585.505373] Lustre: lustre-MDT0000: new disk, initializing [16585.655863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16585.694333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16590.973203] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16602.698773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16602.797504] Lustre: 523033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16602.862113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16602.867682] Lustre: Skipped 1 previous similar message [16602.974420] Lustre: lustre-MDT0001: new disk, initializing [16603.220562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16603.286326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16603.297493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16607.842811] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16618.243258] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16625.901090] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16635.199292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16635.592717] Lustre: lustre-OST0000: new disk, initializing [16635.597099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16635.605312] Lustre: 525004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16635.738852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16641.143730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16641.167473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16641.315918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16642.292549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [16653.107531] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16658.599326] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16658.856415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16664.109369] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16664.298601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16667.106662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16667.108511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16667.141506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16667.159712] Lustre: Skipped 1 previous similar message [16672.231223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16672.241310] Lustre: Skipped 1 previous similar message [16672.781665] Lustre: server umount lustre-OST0000 complete [16677.348309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16677.365656] Lustre: Skipped 1 previous similar message [16677.372454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16677.386635] Lustre: Skipped 1 previous similar message [16682.466639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16682.481295] Lustre: Skipped 1 previous similar message [16683.437553] Lustre: server umount lustre-MDT0000 complete [16687.261323] LustreError: 523025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652790 with bad export cookie 3546838588633732824 [16687.266104] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16687.275224] LustreError: 523025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16687.735040] Lustre: server umount lustre-MDT0001 complete [16714.810691] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [16727.530361] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [16762.285045] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16403 sec ========= 20:01:02 (1788652862) [16764.215537] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:01:04 (1788652864) === [16767.908578] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:01:08 (1788652868) === [16809.107800] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [16812.085968] Key type lgssc unregistered [16812.511266] LNet: 531662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16812.533645] LNetError: 531662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16812.553902] LNet: Removed LNI 192.168.204.109@tcp [16813.566217] Key type .llcrypt unregistered [16813.568109] Key type ._llcrypt unregistered