[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 517424003 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002384] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007499] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021863] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026590] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030039] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.032501] *** VALIDATE proc *** [ 0.033251] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042738] debug: unmapping init [mem 0xffffffffba259000-0xffffffffba260fff] [ 0.045191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054305] rcu: Hierarchical SRCU implementation. [ 0.056567] smp: Bringing up secondary CPUs ... [ 0.057627] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.061310] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136109] node 0 deferred pages initialised in 70ms [ 0.139156] devtmpfs: initialized [ 0.140239] x86/mm: Memory block size: 128MB [ 0.142881] gcov: version magic: 0x41383552 [ 0.144271] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146263] pinctrl core: initialized pinctrl subsystem [ 0.147216] [ 0.147859] ************************************************************* [ 0.148014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149015] ** ** [ 0.150018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151013] ** ** [ 0.152019] ** This means that this kernel is built to expose internal ** [ 0.153014] ** IOMMU data structures, which may compromise security on ** [ 0.154016] ** your system. ** [ 0.155013] ** ** [ 0.156016] ** If you see this message and you are not debugging the ** [ 0.157017] ** kernel, report this immediately to your vendor! ** [ 0.158017] ** ** [ 0.159020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160018] ************************************************************* [ 0.161672] NET: Registered protocol family 16 [ 0.162420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166064] cpuidle: using governor menu [ 0.168816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171551] PCI: Using configuration type 1 for base access [ 0.173130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190137] cryptd: max_cpu_qlen set to 1000 [ 0.193324] ACPI: Added _OSI(Module Device) [ 0.195015] ACPI: Added _OSI(Processor Device) [ 0.197016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205212] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213601] ACPI: Interpreter enabled [ 0.214055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215000] ACPI: Using IOAPIC for interrupt routing [ 0.217110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231851] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245147] acpiphp: Slot [2] registered [ 0.246112] acpiphp: Slot [5] registered [ 0.247239] acpiphp: Slot [6] registered [ 0.249117] acpiphp: Slot [7] registered [ 0.251114] acpiphp: Slot [8] registered [ 0.252125] acpiphp: Slot [9] registered [ 0.254112] acpiphp: Slot [10] registered [ 0.255129] acpiphp: Slot [3] registered [ 0.256102] acpiphp: Slot [4] registered [ 0.258093] acpiphp: Slot [11] registered [ 0.260087] acpiphp: Slot [12] registered [ 0.261092] acpiphp: Slot [13] registered [ 0.263114] acpiphp: Slot [14] registered [ 0.264097] acpiphp: Slot [15] registered [ 0.266156] acpiphp: Slot [16] registered [ 0.267113] acpiphp: Slot [17] registered [ 0.269110] acpiphp: Slot [18] registered [ 0.271135] acpiphp: Slot [19] registered [ 0.272112] acpiphp: Slot [20] registered [ 0.274164] acpiphp: Slot [21] registered [ 0.276102] acpiphp: Slot [22] registered [ 0.277153] acpiphp: Slot [23] registered [ 0.279113] acpiphp: Slot [24] registered [ 0.281134] acpiphp: Slot [25] registered [ 0.282109] acpiphp: Slot [26] registered [ 0.284147] acpiphp: Slot [27] registered [ 0.286137] acpiphp: Slot [28] registered [ 0.288113] acpiphp: Slot [29] registered [ 0.289120] acpiphp: Slot [30] registered [ 0.291119] acpiphp: Slot [31] registered [ 0.293092] PCI host bridge to bus 0000:00 [ 0.294021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313247] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317425] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326801] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332639] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335052] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337044] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.346871] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.352826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377895] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434852] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462047] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.518723] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.561018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576404] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.599024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.621029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.639815] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.652022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.724099] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.733022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.744029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.779186] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782417] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785408] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.787439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.790284] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.795170] iommu: Default domain type: Passthrough [ 0.796698] SCSI subsystem initialized [ 0.799209] ACPI: bus type USB registered [ 0.800238] usbcore: registered new interface driver usbfs [ 0.803126] usbcore: registered new interface driver hub [ 0.804390] usbcore: registered new device driver usb [ 0.807239] pps_core: LinuxPPS API ver. 1 registered [ 0.809018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.814203] PTP clock support registered [ 0.818301] EDAC MC: Ver: 3.0.0 [ 0.821186] PCI: Using ACPI for IRQ routing [ 0.823944] NetLabel: Initializing [ 0.825018] NetLabel: domain hash size = 128 [ 0.826015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.828099] NetLabel: unlabeled traffic allowed by default [ 0.830196] vgaarb: loaded [ 0.832318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.838425] clocksource: Switched to clocksource kvm-clock [ 0.954711] VFS: Disk quotas dquot_6.6.0 [ 0.957084] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.960353] *** VALIDATE ramfs *** [ 0.961841] *** VALIDATE hugetlbfs *** [ 0.963727] pnp: PnP ACPI init [ 0.966834] pnp: PnP ACPI: found 6 devices [ 1.024853] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.028638] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.030867] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.033160] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.035709] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.038443] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.041287] NET: Registered protocol family 2 [ 1.043900] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.048549] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.052466] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.058675] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.063133] TCP: Hash tables configured (established 65536 bind 65536) [ 1.066265] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.069520] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.071906] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.074479] NET: Registered protocol family 1 [ 1.077650] RPC: Registered named UNIX socket transport module. [ 1.080562] RPC: Registered udp transport module. [ 1.082702] RPC: Registered tcp transport module. [ 1.084634] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.087555] NET: Registered protocol family 44 [ 1.089562] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.092109] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.094416] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.096893] PCI: CLS 0 bytes, default 64 [ 1.098545] Unpacking initramfs... [ 2.653591] debug: unmapping init [mem 0xffff9b123cc54000-0xffff9b123ffbffff] [ 2.664837] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.670872] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.680859] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.188343] Initialise system trusted keyrings [ 3.190401] Key type blacklist registered [ 3.192592] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.204476] zbud: loaded [ 3.209290] *** VALIDATE nfs *** [ 3.210725] *** VALIDATE nfs4 *** [ 3.212661] pstore: using deflate compression [ 3.216874] Platform Keyring initialized [ 3.330345] NET: Registered protocol family 38 [ 3.332062] Key type asymmetric registered [ 3.333434] Asymmetric key parser 'x509' registered [ 3.335185] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.339041] io scheduler mq-deadline registered [ 3.340677] io scheduler kyber registered [ 3.342349] io scheduler bfq registered [ 3.343996] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.346083] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.348655] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.351392] ACPI: Power Button [PWRF] [ 3.356295] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.363145] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.388383] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.395798] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.421792] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.450804] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.477130] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.481842] Non-volatile memory driver v1.3 [ 3.484365] Linux agpgart interface v0.103 [ 3.522211] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.527377] vda: detected capacity change from 0 to 76779520 [ 3.546142] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.549569] vdb: detected capacity change from 0 to 1073741824 [ 3.567380] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.570492] vdc: detected capacity change from 0 to 2621440000 [ 3.586135] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.589410] vdd: detected capacity change from 0 to 2621440000 [ 3.610142] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.612388] vde: detected capacity change from 0 to 4294967296 [ 3.632724] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.635435] vdf: detected capacity change from 0 to 4294967296 [ 3.643762] libphy: Fixed MDIO Bus: probed [ 3.649865] usbcore: registered new interface driver usbserial_generic [ 3.652333] usbserial: USB Serial support registered for generic [ 3.654863] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.659305] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.661276] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.663712] mousedev: PS/2 mouse device common for all mice [ 3.666693] rtc_cmos 00:05: RTC can wake from S4 [ 3.669343] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.670673] rtc_cmos 00:05: registered as rtc0 [ 3.676353] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.676465] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.681099] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.683315] intel_pstate: CPU model not supported [ 3.689850] hid: raw HID events driver (C) Jiri Kosina [ 3.692494] usbcore: registered new interface driver usbhid [ 3.695077] usbhid: USB HID core driver [ 3.696943] drop_monitor: Initializing network drop monitor service [ 3.700120] Initializing XFRM netlink socket [ 3.702455] NET: Registered protocol family 10 [ 3.705634] Segment Routing with IPv6 [ 3.707262] NET: Registered protocol family 17 [ 3.709882] mpls_gso: MPLS GSO support [ 3.717978] RAS: Correctable Errors collector initialized. [ 3.720340] AVX version of gcm_enc/dec engaged. [ 3.722046] AES CTR mode by8 optimization enabled [ 3.800152] sched_clock: Marking stable (3800119678, 0)->(4761341967, -961222289) [ 3.803880] registered taskstats version 1 [ 3.806294] Loading compiled-in X.509 certificates [ 3.808392] zswap: loaded using pool lzo/zbud [ 3.837408] Key type big_key registered [ 3.851860] Key type encrypted registered [ 3.853846] ima: No TPM chip found, activating TPM-bypass! [ 3.856362] ima: Allocated hash algorithm: sha1 [ 3.858428] ima: No architecture policies found [ 3.860603] evm: Initialising EVM extended attributes: [ 3.862896] evm: security.selinux [ 3.864426] evm: security.ima [ 3.865751] evm: security.capability [ 3.867447] evm: HMAC attrs: 0x1 [ 3.870275] rtc_cmos 00:05: setting system clock to 2026-09-08 05:56:22 UTC (1788846982) [ 3.879190] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.882797] debug: unmapping init [mem 0xffffffffb9f82000-0xffffffffba258fff] [ 3.892075] Write protecting the kernel read-only data: 28672k [ 3.896060] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 3.899919] debug: unmapping init [mem 0xffffffffb8f14000-0xffffffffb8ffffff] [ 3.937788] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.945814] systemd[1]: Detected virtualization kvm. [ 3.947712] systemd[1]: Detected architecture x86-64. [ 3.949536] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.974281] systemd[1]: No hostname configured. [ 3.976337] systemd[1]: Set hostname to . [ 3.978688] random: systemd: uninitialized urandom read (16 bytes read) [ 3.981482] systemd[1]: Initializing machine ID from random generator. [ 4.046922] random: ln: uninitialized urandom read (6 bytes read) [ 4.140220] random: systemd: uninitialized urandom read (16 bytes read) [ 4.143196] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.148777] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.153193] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.811406] device-mapper: uevent: version 1.0.3 [ 4.813302] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.543810] virtio_net virtio0 ens2: renamed from eth0 [ 5.563580] scsi host0: ata_piix [ 5.566512] random: fast init done [ 5.595333] scsi host1: ata_piix [ 5.597061] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.599674] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.438989] random: crng init done [ 10.444198] random: 7 urandom warning(s) missed due to ratelimiting [ 11.161042] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.248920] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.792614] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.839494] SELinux: Disabled at runtime. [ 18.038202] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.067714] systemd[1]: Detected virtualization kvm. [ 18.073906] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.139718] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.150587] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.180513] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.204535] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.212827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.255042] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.275977] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 20.740778] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.195266] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.442276] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.529685] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.288617] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.397623] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.576536] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 30.391701] NFS: Registering the id_resolver key type [ 30.396385] Key type id_resolver registered [ 30.400892] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg419-server login: [ 90.030194] libcfs: loading out-of-tree module taints kernel. [ 90.107569] Key type ._llcrypt registered [ 90.109903] Key type .llcrypt registered [ 90.259705] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 115.213857] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 117.247681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.288300] alg: No test for adler32 (adler32-zlib) [ 119.134724] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 120.344321] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 122.272233] Key type lgssc registered [ 124.985644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.531271] hrtimer: interrupt took 4464505 ns [ 149.289908] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 201.836823] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 216.878383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 216.942372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.319930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.381495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.540842] Lustre: lustre-MDT0000: new disk, initializing [ 218.688150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.736636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.158950] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.095290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.221406] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 241.266938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 241.276694] Lustre: Skipped 1 previous similar message [ 241.431588] Lustre: lustre-MDT0001: new disk, initializing [ 241.562179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.591790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.610136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 247.271811] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.281222] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 265.641030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.113432] Lustre: lustre-OST0000: new disk, initializing [ 266.117473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 266.121560] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.232126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.717866] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.032588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 275.046624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 275.145031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 292.519322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.764485] Lustre: lustre-OST0001: new disk, initializing [ 292.767591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 292.772815] Lustre: 9543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.841028] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.095906] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.600931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.608469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.686054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.317212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.330247] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 328.643579] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing check_logdir /tmp/testlogs/ [ 333.818451] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing yml_node [ 339.205880] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 342.796567] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 346.349145] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 348.430676] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 02:02:05 EDT 2026 [ 368.546596] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 370.183119] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 388.581617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.582505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.599744] Lustre: Skipped 3 previous similar messages [ 388.619023] Lustre: Skipped 2 previous similar messages [ 393.698100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 393.708144] Lustre: Skipped 3 previous similar messages [ 398.817696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.849087] Lustre: Skipped 3 previous similar messages [ 401.376158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 401.492733] Lustre: server umount lustre-MDT0000 complete [ 406.502680] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847385 with bad export cookie 16730856958736982637 [ 406.525354] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.529559] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 409.057086] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.062037] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.069419] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 409.072351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 409.083733] Lustre: Skipped 1 previous similar message [ 409.099396] Lustre: Skipped 1 previous similar message [ 411.105138] LustreError: 7936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.122217] LustreError: 7936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 412.956175] Lustre: server umount lustre-MDT0001 complete [ 417.824740] Lustre: server umount lustre-OST0000 complete [ 422.866835] Lustre: server umount lustre-OST0001 complete [ 429.959211] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 443.557178] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 486.542548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.851403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 486.907281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 487.078703] Lustre: lustre-MDT0000: new disk, initializing [ 487.256663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.281237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 491.967562] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 503.370700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.556974] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 503.684984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 503.688356] Lustre: Skipped 1 previous similar message [ 503.805772] Lustre: lustre-MDT0001: new disk, initializing [ 503.939341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 503.988057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 504.009172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 509.086850] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 519.324537] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.577255] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 533.296678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.722490] Lustre: lustre-OST0000: new disk, initializing [ 533.732761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 533.753275] Lustre: 17070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 533.967423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.765919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 535.785966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 535.935242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 541.194437] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 554.121579] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.148840] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 560.408643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 564.982377] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 565.174733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 571.881651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.885880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.912818] LustreError: Skipped 1 previous similar message [ 571.919872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.929872] Lustre: Skipped 3 previous similar messages [ 573.503185] Lustre: server umount lustre-OST0000 complete [ 580.580100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.597100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.614508] Lustre: Skipped 1 previous similar message [ 580.631044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.105225] Lustre: server umount lustre-MDT0000 complete [ 587.234779] LustreError: 15115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.262439] LustreError: 15115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 590.866995] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847569 with bad export cookie 16730856958736984170 [ 590.883478] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.645919] Lustre: server umount lustre-MDT0001 complete [ 596.531758] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:06:13 (1788847573) [ 600.034488] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 601.876466] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:06:18 (1788847578) [ 608.830974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.437514] LustreError: 19481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.530870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.234949] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 614.886798] LustreError: 19482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.741813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.164778] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 642.300953] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.978519] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.149445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 663.704942] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 671.914879] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 694.244785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.262432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.282524] Lustre: Skipped 1 previous similar message [ 694.291934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.296085] Lustre: Skipped 2 previous similar messages [ 697.100741] Lustre: server umount lustre-OST0000 complete [ 706.018773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.044466] Lustre: Skipped 1 previous similar message [ 707.810743] Lustre: server umount lustre-MDT0000 complete [ 711.145341] LustreError: 19481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.171380] LustreError: 19481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 712.587694] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847691 with bad export cookie 16730856958736985052 [ 712.592155] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.600486] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 712.993731] Lustre: server umount lustre-MDT0001 complete [ 726.792733] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 729.884213] Key type lgssc unregistered [ 730.280890] LNet: 23046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.288791] LNetError: 23046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.300445] LNet: Removed LNI 192.168.204.119@tcp [ 731.490177] Key type .llcrypt unregistered [ 731.492661] Key type ._llcrypt unregistered [ 757.081346] Key type ._llcrypt registered [ 757.086873] Key type .llcrypt registered [ 757.172576] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 758.342878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.384234] alg: No test for adler32 (adler32-zlib) [ 759.439919] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 759.758561] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 761.425310] Key type lgssc registered [ 762.572557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 771.648958] loop: module loaded [ 771.779981] loop0: detected capacity change from 0 to 204800000 [ 772.157895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.173723] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.388321] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.317169] loop0: detected capacity change from 0 to 204800000 [ 778.500226] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.515614] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.533588] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 783.763499] loop0: detected capacity change from 0 to 204800000 [ 788.771653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.784388] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.042609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 790.151707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 790.261345] Lustre: lustre-MDT0000: new disk, initializing [ 790.367918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.388653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 794.863557] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 803.279154] loop1: detected capacity change from 0 to 204800000 [ 808.965711] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 809.236861] Lustre: lustre-OST07c6: new disk, initializing [ 809.249130] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 809.259347] Lustre: Skipped 1 previous similar message [ 809.267615] Lustre: 26232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 809.379430] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 815.473313] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 816.877781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 816.888794] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 816.944099] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 820.639840] Lustre: 25220:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 820.651549] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 820.658664] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 820.664403] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 820.670579] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 820.676787] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 835.932582] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 835.949991] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 835.965702] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 835.978685] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 835.987192] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 835.996582] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 841.158541] Lustre: 26806:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 841.171671] Lustre: 26806:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 841.177636] Lustre: 26806:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 841.186320] Lustre: 26806:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 841.203327] Lustre: 26806:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 841.212555] Lustre: 26806:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 841.221639] Lustre: 26806:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 841.230605] Lustre: 26806:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 841.240352] Lustre: 26806:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 841.252168] Lustre: 26806:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 841.261924] Lustre: 26806:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 841.266390] Lustre: 26806:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.442716] Lustre: 25221:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 844.461174] Lustre: 25221:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 844.469800] Lustre: 25221:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 844.483725] Lustre: 25221:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.491918] Lustre: 25221:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 844.503483] Lustre: 25221:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.513157] Lustre: 25221:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 844.522783] Lustre: 25221:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.528672] Lustre: 25221:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 844.538277] Lustre: 25221:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.546471] Lustre: 25221:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 844.552390] Lustre: 25221:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 849.694840] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 849.707369] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 849.717561] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 849.725368] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 849.744140] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 849.751681] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 849.762930] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 849.769149] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 849.774302] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 849.778657] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 849.790781] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 849.799486] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.216453] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 858.221827] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 858.226441] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 858.233208] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 858.239019] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 858.248880] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 858.260975] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 858.266546] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 858.270208] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 858.275479] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 858.279689] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 858.284040] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 914.403039] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.405208] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 914.433845] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 915.892592] Lustre: server umount lustre-OST07c6 complete [ 925.040381] Lustre: server umount lustre-MDT0000 complete [ 943.390803] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 946.342909] Key type lgssc unregistered [ 946.599753] LNet: 28786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.618866] LNetError: 28786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.692892] LNet: Removed LNI 192.168.204.119@tcp [ 948.601551] Key type .llcrypt unregistered [ 948.604175] Key type ._llcrypt unregistered [ 967.294913] Key type ._llcrypt registered [ 967.297601] Key type .llcrypt registered [ 967.490202] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 969.242276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.277162] alg: No test for adler32 (adler32-zlib) [ 970.249846] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 970.546398] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 972.288246] Key type lgssc registered [ 973.567308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 982.604034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.636416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.454093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.420687] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 994.794789] LustreError: 30053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.818199] LustreError: 30053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 999.791355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.915149] LustreError: 30054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.272918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.259814] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1014.465705] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1021.118178] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1028.200217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1028.747704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.765785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1035.156451] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1041.910396] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1064.387574] Lustre: server umount lustre-OST0000 complete [ 1071.078932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.100353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1071.108734] Lustre: Skipped 1 previous similar message [ 1071.125669] Lustre: Skipped 1 previous similar message [ 1076.197452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.218430] Lustre: Skipped 1 previous similar message [ 1079.400182] Lustre: server umount lustre-MDT0000 complete [ 1081.320143] LustreError: 30782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.342070] LustreError: 30782:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1082.337082] Lustre: 29364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848044/real 1788848044] req@00000000dc8759af x1875743027003008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788848060 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1082.350653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.112433] LustreError: 30789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848062 with bad export cookie 2733247689648288712 [ 1084.117202] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.133899] LustreError: 30789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.641877] Lustre: server umount lustre-MDT0001 complete [ 1097.616672] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1101.082946] Key type lgssc unregistered [ 1101.429834] LNet: 33790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.456138] LNetError: 33790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1101.481800] LNet: Removed LNI 192.168.204.119@tcp [ 1102.442207] Key type .llcrypt unregistered [ 1102.445784] Key type ._llcrypt unregistered [ 1122.426770] Key type ._llcrypt registered [ 1122.432525] Key type .llcrypt registered [ 1122.558550] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1124.628812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.714388] alg: No test for adler32 (adler32-zlib) [ 1125.796426] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1126.064567] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1127.761128] Key type lgssc registered [ 1129.684135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.051190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.094775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.972742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.097725] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1151.463357] LustreError: 35058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.478529] LustreError: 35058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1156.592298] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.663644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.050494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1161.332681] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1168.764995] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1174.690809] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1181.160910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1181.519725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.685340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1187.497589] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1195.331278] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1210.855269] Lustre: server umount lustre-OST0000 complete [ 1217.508866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.519097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.531568] Lustre: Skipped 1 previous similar message [ 1217.550538] Lustre: Skipped 1 previous similar message [ 1222.631671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.645995] Lustre: Skipped 1 previous similar message [ 1225.848605] Lustre: server umount lustre-MDT0000 complete [ 1227.751916] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.779694] LustreError: 35059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1228.772213] Lustre: 34369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848191/real 1788848191] req@000000004de8ecd8 x1875743189529216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788848207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1228.801383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.858283] Lustre: 34369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848192/real 1788848192] req@0000000053ce9baf x1875743189529600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788848208 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1230.226589] LustreError: 35039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848208 with bad export cookie 10192809957017912166 [ 1230.235689] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.246307] LustreError: 35039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1230.896816] Lustre: server umount lustre-MDT0001 complete [ 1243.809987] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1247.469917] Key type lgssc unregistered [ 1247.845470] LNet: 38606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.854052] LNetError: 38606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.883930] LNet: Removed LNI 192.168.204.119@tcp [ 1248.744158] Key type .llcrypt unregistered [ 1248.748868] Key type ._llcrypt unregistered [ 1276.370768] Key type ._llcrypt registered [ 1276.372884] Key type .llcrypt registered [ 1276.499646] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1277.857291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.906132] alg: No test for adler32 (adler32-zlib) [ 1278.971539] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1279.418606] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1281.152252] Key type lgssc registered [ 1282.381917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1290.952229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1290.975179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.413724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.821489] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1303.012215] LustreError: 40023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.030114] LustreError: 40023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1308.132469] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.355103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.082243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1314.738333] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1323.262215] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.979630] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.771951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.191554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.238471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1343.785473] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1351.779259] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1363.789066] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 1364.971934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.973926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.005976] Lustre: Skipped 1 previous similar message [ 1365.025258] Lustre: Skipped 2 previous similar messages [ 1367.163417] Lustre: server umount lustre-MDT0000 complete [ 1368.875292] LustreError: 40018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.040881] LustreError: 41867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848349 with bad export cookie 8625672840101662412 [ 1371.051733] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.022328] LustreError: 42515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.042107] LustreError: 42515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1374.058820] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 1375.209826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.234404] Lustre: Skipped 1 previous similar message [ 1376.225726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1376.232077] Lustre: Skipped 1 previous similar message [ 1377.672243] Lustre: server umount lustre-MDT0001 complete [ 1393.347843] Lustre: server umount lustre-OST0000 complete [ 1413.625468] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1416.868605] Key type lgssc unregistered [ 1417.221834] LNet: 43881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.234927] LNetError: 43881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.261086] LNet: Removed LNI 192.168.204.119@tcp [ 1418.235474] Key type .llcrypt unregistered [ 1418.241930] Key type ._llcrypt unregistered [ 1445.009840] Key type ._llcrypt registered [ 1445.015611] Key type .llcrypt registered [ 1445.126335] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1446.854515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.051397] alg: No test for adler32 (adler32-zlib) [ 1448.494381] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1448.886575] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1450.657604] Key type lgssc registered [ 1452.258991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.327232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1462.365482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.826311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.537365] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1474.028831] LustreError: 45297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.044101] LustreError: 45297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1479.139032] LustreError: 45298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.204204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.715235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1487.679740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1497.277512] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1504.654799] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1512.946352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.380849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.592623] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1532.046635] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1533.955473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1543.999987] Lustre: lustre-OST0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 1544.164667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.167128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.185940] Lustre: Skipped 1 previous similar message [ 1549.099635] Lustre: lustre-OST0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 1549.113178] Lustre: Skipped 2 previous similar messages [ 1549.798845] Lustre: server umount lustre-OST0000 complete [ 1562.324967] LustreError: 45277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.19@tcp arrived at 1788848540 with bad export cookie 7646747386894059648 [ 1571.554942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.578987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.591570] Lustre: Skipped 1 previous similar message [ 1571.618981] Lustre: Skipped 3 previous similar messages [ 1574.692781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.720460] Lustre: Skipped 1 previous similar message [ 1578.178889] Lustre: server umount lustre-MDT0000 complete [ 1580.021789] LustreError: 47136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.049249] LustreError: 47136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1582.924674] LustreError: 45279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848561 with bad export cookie 7646747386894058983 [ 1582.934694] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.937128] LustreError: 45279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1589.597320] Lustre: server umount lustre-MDT0001 complete [ 1602.341570] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1605.342081] Key type lgssc unregistered [ 1605.725909] LNet: 49011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.735879] LNetError: 49011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.755069] LNet: Removed LNI 192.168.204.119@tcp [ 1606.851441] Key type .llcrypt unregistered [ 1606.855294] Key type ._llcrypt unregistered [ 1630.424385] Key type ._llcrypt registered [ 1630.426730] Key type .llcrypt registered [ 1630.541614] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1631.940943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.997362] alg: No test for adler32 (adler32-zlib) [ 1633.060598] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1633.291908] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1635.012838] Key type lgssc registered [ 1636.410788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.026779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.037354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.531058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1650.214577] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1655.789307] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.805557] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1658.544932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.983963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.393178] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1671.119081] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.508151] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.707710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.170971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.283236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1695.042372] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1703.263271] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.216144] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1712.320959] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1724.452831] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 2h25m01s [ 1726.276525] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1730.531383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.535346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.547867] Lustre: Skipped 2 previous similar messages [ 1730.550631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 127.0.0.2@tcp failed: rc = -107 [ 1730.554822] Lustre: Skipped 2 previous similar messages [ 1731.619526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.643633] Lustre: Skipped 3 previous similar messages [ 1734.457497] Lustre: server umount lustre-MDT0000 complete [ 1737.008495] LustreError: 50420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.402404] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1743.153533] LustreError: 50421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.186189] LustreError: 50421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1744.726740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.908629] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.124992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.184523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1746.163738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1749.360043] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1752.702934] Lustre: DEBUG MARKER: Wait for df (36721) ... [ 1754.512056] Lustre: DEBUG MARKER: done [ 1760.749837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1760.761724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.779680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.785287] Lustre: Skipped 1 previous similar message [ 1765.573503] Lustre: server umount lustre-OST0000 complete [ 1772.002835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.004928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.017050] Lustre: Skipped 2 previous similar messages [ 1772.030584] Lustre: Skipped 3 previous similar messages [ 1775.652499] Lustre: server umount lustre-MDT0000 complete [ 1779.171226] LustreError: 51153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.184716] LustreError: 51153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1780.102370] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848758 with bad export cookie 2777535188827696704 [ 1780.103598] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.113232] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1780.485453] Lustre: server umount lustre-MDT0001 complete [ 1791.596653] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1794.360908] Key type lgssc unregistered [ 1794.684237] LNet: 55348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.694485] LNetError: 55348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.710798] LNet: Removed LNI 192.168.204.119@tcp [ 1795.706531] Key type .llcrypt unregistered [ 1795.708744] Key type ._llcrypt unregistered [ 1836.923312] Key type ._llcrypt registered [ 1836.926838] Key type .llcrypt registered [ 1837.047573] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1838.166781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.328814] alg: No test for adler32 (adler32-zlib) [ 1839.474569] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 1839.842723] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1841.528692] Key type lgssc registered [ 1842.922233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.259545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.302374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.642787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.665938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.815498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.058657] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1867.594182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.711581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1867.797272] Lustre: 57553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1868.370296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.202638] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1882.971512] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1889.765164] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1897.010882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1897.241981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1897.423832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.299803] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1904.315371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1912.413380] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1918.442626] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 2h28m15s [ 1920.112223] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1927.696837] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1930.789817] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1931.018362] LustreError: 58695:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1951.099713] Lustre: lustre-MDT0000: Client a5a0a226-a9a0-4b67-80e4-be5dd2a3f717 (at 192.168.204.19@tcp) reconnecting [ 1971.113327] LustreError: 58695:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1973.413436] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1975.700174] Lustre: DEBUG MARKER: done [ 1986.024266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.046166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.060377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.530872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1991.652749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.657576] Lustre: Skipped 2 previous similar messages [ 1996.257924] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1996.505516] Lustre: server umount lustre-OST0000 complete [ 2001.377096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2001.388505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.405809] Lustre: Skipped 1 previous similar message [ 2001.414793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.280644] Lustre: server umount lustre-MDT0000 complete [ 2009.712782] LustreError: 57546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848988 with bad export cookie 11515801833830271270 [ 2009.719855] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.732602] LustreError: 57546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2010.141461] Lustre: server umount lustre-MDT0001 complete [ 2021.990766] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2024.939995] Key type lgssc unregistered [ 2025.367808] LNet: 61882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.377127] LNetError: 61882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.393086] LNet: Removed LNI 192.168.204.119@tcp [ 2026.284139] Key type .llcrypt unregistered [ 2026.285783] Key type ._llcrypt unregistered [ 2069.829974] Key type ._llcrypt registered [ 2069.840635] Key type .llcrypt registered [ 2069.950366] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2070.811950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2070.825121] alg: No test for adler32 (adler32-zlib) [ 2071.833710] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2072.011232] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2073.664656] Key type lgssc registered [ 2074.500470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2077.333087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2077.351689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.637499] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2078.656447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2078.753844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.575737] Lustre: Failing over lustre-MDT0000 [ 2079.776557] Lustre: server umount lustre-MDT0000 complete [ 2086.514259] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:31:03 (1788849063) [ 2092.771379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.108980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.081894] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2105.950365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.027815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2106.059118] Lustre: 64677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2106.298137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.613303] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2118.743877] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.924964] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.359634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2131.489152] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2131.631553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.002205] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2140.767777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2143.229432] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2147.734934] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2155.000356] Lustre: server umount lustre-MDT0000 complete [ 2156.001663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.022082] Lustre: Skipped 1 previous similar message [ 2159.098663] LustreError: 64668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849137 with bad export cookie 3146487153663192798 [ 2159.115096] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.528349] Lustre: server umount lustre-MDT0001 complete [ 2163.854385] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2169.386933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2177.250473] Lustre: 63599:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849139/real 1788849139] req@00000000827bcff7 x1875744181490304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.268574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.273836] Lustre: Skipped 1 previous similar message [ 2178.438570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.488936] Lustre: 63601:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849142/real 1788849142] req@00000000ebc0a3ef x1875744181490432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.371341] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849144/real 1788849144] req@00000000d7918c3a x1875744181490560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.799869] LustreError: 66523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.819520] LustreError: 66523:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2183.881568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.977943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2184.865132] Lustre: 63598:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849147/real 1788849147] req@00000000d247d186 x1875744181490816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2186.980388] LustreError: 68444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.991167] LustreError: 68444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2188.113743] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2191.076096] Lustre: 63598:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849153/real 1788849153] req@0000000059e1e2e4 x1875744181490944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2192.368972] LustreError: 68442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.384355] LustreError: 68442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2195.495215] LustreError: 68443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.503301] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2195.516116] LustreError: 68443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2196.389544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.814527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.962966] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2202.089512] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2208.395627] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.636174] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.570698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.578346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2222.580855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.585800] Lustre: Skipped 1 previous similar message [ 2226.887745] Lustre: server umount lustre-MDT0000 complete [ 2227.687500] LustreError: 68444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.711351] LustreError: 68444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2230.433832] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849209 with bad export cookie 3146487153663195073 [ 2230.446211] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.448994] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2230.782382] Lustre: server umount lustre-MDT0001 complete [ 2237.786700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2247.780530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.184636] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849211/real 1788849211] req@00000000d8c8ffc0 x1875744181517568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788849227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2249.216279] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2249.227764] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.392735] LustreError: 63597:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bdb35496 x1875744181518592/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2255.768756] LustreError: 66523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.846737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.886523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2260.167763] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2268.105710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.151122] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2273.775950] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2277.883542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2279.264063] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2285.100776] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2293.222590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.224179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.233516] Lustre: Skipped 2 previous similar messages [ 2293.247630] Lustre: Skipped 3 previous similar messages [ 2297.710837] Lustre: server umount lustre-MDT0000 complete [ 2298.338032] LustreError: 71125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.352932] LustreError: 71125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2301.655788] LustreError: 70271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849280 with bad export cookie 3146487153663196452 [ 2301.660624] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.670877] LustreError: 70271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2302.246810] Lustre: server umount lustre-MDT0001 complete [ 2308.903954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2312.086070] Lustre: DEBUG MARKER: files compared the same [ 2324.450559] Lustre: server umount lustre-OST0000 complete [ 2340.946958] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2343.658679] Key type lgssc unregistered [ 2344.030957] LNet: 74400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.039251] LNetError: 74400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.067598] LNet: Removed LNI 192.168.204.119@tcp [ 2344.937813] Key type .llcrypt unregistered [ 2344.940285] Key type ._llcrypt unregistered [ 2364.892990] Key type ._llcrypt registered [ 2364.897330] Key type .llcrypt registered [ 2365.012974] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2366.098613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.130963] alg: No test for adler32 (adler32-zlib) [ 2367.250439] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2367.492585] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2369.147676] Key type lgssc registered [ 2370.265163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.841856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.876940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.373337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.482582] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug +malloc all [ 2387.941544] LustreError: 75796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.955128] LustreError: 75796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2390.006136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.312482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2393.967899] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug +malloc all [ 2400.730888] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.391393] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2410.942121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.215125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.369812] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug +malloc all [ 2422.346404] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.513723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2430.945733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.951241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.962611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2432.995460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.997341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2433.016289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2433.022758] Lustre: Skipped 1 previous similar message [ 2438.114142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2438.121462] Lustre: Skipped 1 previous similar message [ 2443.232374] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2443.235089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2443.260139] Lustre: Skipped 1 previous similar message [ 2443.404430] Lustre: server umount lustre-OST0000 complete [ 2448.353805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.370404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.378888] Lustre: Skipped 1 previous similar message [ 2453.444963] Lustre: server umount lustre-MDT0000 complete [ 2453.476465] LustreError: 75796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.511802] LustreError: 75796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2456.932210] LustreError: 78575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849435 with bad export cookie 17894104986683140542 [ 2456.937978] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.942733] LustreError: 78575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2457.334195] Lustre: server umount lustre-MDT0001 complete [ 2467.389570] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2469.974918] Key type lgssc unregistered [ 2470.224223] LNet: 79325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.232991] LNetError: 79325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.252182] LNet: Removed LNI 192.168.204.119@tcp [ 2470.949266] Key type .llcrypt unregistered [ 2470.953521] Key type ._llcrypt unregistered [ 2492.933479] Key type ._llcrypt registered [ 2492.942207] Key type .llcrypt registered [ 2493.079128] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2494.235234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.245395] alg: No test for adler32 (adler32-zlib) [ 2495.249550] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2495.460287] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2497.160606] Key type lgssc registered [ 2498.069331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2505.345869] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2505.363305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.866227] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2525.986071] LustreError: 80712:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 2552.608210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.170686] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2562.782806] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2568.614957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.716447] Lustre: *** cfs_fail_loc=706, val=0*** [ 2568.987251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.033661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2571.760461] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2574.305356] LustreError: 80737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.336879] LustreError: 80737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2578.323386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.596449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2582.146817] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2588.172886] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2592.523312] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.281476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.291243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.310200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.790396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.796949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.812158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.370605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.233776] Lustre: server umount lustre-OST0000 complete [ 2603.490056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.493581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.496129] Lustre: Skipped 1 previous similar message [ 2603.504759] Lustre: Skipped 2 previous similar messages [ 2608.610695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.617566] Lustre: Skipped 1 previous similar message [ 2609.336793] Lustre: server umount lustre-MDT0000 complete [ 2612.514232] LustreError: 82537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849591 with bad export cookie 10491994314705569399 [ 2612.522066] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.528443] LustreError: 82537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2612.773419] Lustre: server umount lustre-MDT0001 complete [ 2621.295197] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2623.758385] Key type lgssc unregistered [ 2624.039468] LNet: 84324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.048552] LNetError: 84324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.063264] LNet: Removed LNI 192.168.204.119@tcp [ 2624.867971] Key type .llcrypt unregistered [ 2624.870590] Key type ._llcrypt unregistered [ 2644.977727] Key type ._llcrypt registered [ 2644.979928] Key type .llcrypt registered [ 2645.079736] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2645.729952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.824487] alg: No test for adler32 (adler32-zlib) [ 2646.850363] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2647.050574] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2648.704339] Key type lgssc registered [ 2649.627545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2655.677789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.695188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.300445] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2666.615579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.041377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.518258] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2672.099258] LustreError: 86403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.111886] LustreError: 86403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2677.220070] LustreError: 86402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.241605] LustreError: 86402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2677.416346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.734577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.459423] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2682.850431] LustreError: 86407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.872680] LustreError: 86407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2687.970992] LustreError: 86402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.992899] LustreError: 86402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2688.978281] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.986318] LustreError: 85072:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eb200e74 x1875744784393088/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2689.006141] Lustre: MGS: Client f1f049d4-261a-4d5c-aea5-cc36b09692c9 (at 0@lo) reconnecting [ 2689.034713] LustreError: 86417:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2689.190058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.250186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2692.494689] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2704.867238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.876468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.892834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.941325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2707.948831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.949679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.968884] Lustre: Skipped 1 previous similar message [ 2710.361204] Lustre: server umount lustre-OST0000 complete [ 2718.177389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2718.179727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.186106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.199255] Lustre: Skipped 1 previous similar message [ 2718.206936] Lustre: Skipped 1 previous similar message [ 2719.428054] Lustre: server umount lustre-MDT0000 complete [ 2722.930295] Lustre: server umount lustre-MDT0001 complete [ 2726.263689] Lustre: server umount MGS complete [ 2735.923147] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:41:53 (1788849713) [ 2757.758029] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 2764.298977] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2791.206858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.713491] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2802.342038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.502973] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2802.541150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2802.629697] Lustre: lustre-MDT0001: new disk, initializing [ 2802.701596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.734144] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2802.741657] Lustre: 93567:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2802.750351] LustreError: 93567:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2805.970983] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2814.010643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.183623] Lustre: lustre-OST0000: new disk, initializing [ 2814.186397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2814.192052] Lustre: 94332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2814.272320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.204697] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2821.160632] Lustre: 94751:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004221bb44 x1875744784422912/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788849815 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2826.892423] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.948576] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.972822] Lustre: MGS: Client 957267ac-aded-492f-98d4-ee7f68d2c98c (at 0@lo) reconnecting [ 2832.035786] Lustre: 95377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2832.060711] LustreError: 93553:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2832.099184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2832.154443] Lustre: lustre-MDT0000: new disk, initializing [ 2832.304311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.320793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2832.427049] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2833.379993] Lustre: 95401:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000265d5e50 x1875744784429312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788849828 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2835.708391] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2841.569652] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2842.476771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2842.486232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2842.628641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2862.573594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.576985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.582114] Lustre: Skipped 1 previous similar message [ 2862.589280] Lustre: Skipped 1 previous similar message [ 2867.684215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.689561] Lustre: Skipped 1 previous similar message [ 2868.626820] Lustre: server umount lustre-OST0000 complete [ 2872.806044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.809722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.814783] Lustre: server umount lustre-MDT0000 complete [ 2877.925622] LustreError: 93537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.933889] LustreError: 93537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2887.029181] Lustre: server umount lustre-MDT0001 complete [ 2890.465296] Lustre: server umount MGS complete [ 2899.862174] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:44:37 (1788849877) [ 2914.159218] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2916.703030] Key type lgssc unregistered [ 2916.970697] LNet: 97902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.978062] LNetError: 97902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.997524] LNet: Removed LNI 192.168.204.119@tcp [ 2917.809768] Key type .llcrypt unregistered [ 2917.811780] Key type ._llcrypt unregistered [ 2931.238701] Key type ._llcrypt registered [ 2931.243269] Key type .llcrypt registered [ 2931.348765] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2932.225190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.249294] alg: No test for adler32 (adler32-zlib) [ 2933.242889] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 2933.457659] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2935.120619] Key type lgssc registered [ 2936.063986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2942.695278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2942.727626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.173562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.769908] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2953.700216] LustreError: 99098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.716465] LustreError: 99098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2954.510534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.795701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.216826] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2963.806640] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2967.631777] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2972.732262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.967966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.330450] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2982.635988] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2989.364262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2999.782654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.786946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.786945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.802017] Lustre: Skipped 1 previous similar message [ 3004.280377] Lustre: server umount lustre-OST0000 complete [ 3010.019027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.023342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.032323] Lustre: Skipped 1 previous similar message [ 3010.047206] Lustre: Skipped 2 previous similar messages [ 3013.465640] Lustre: server umount lustre-MDT0000 complete [ 3015.142764] LustreError: 99095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.163435] LustreError: 99095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3016.548578] LustreError: 99078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849995 with bad export cookie 16129233335617584026 [ 3016.551512] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.559897] LustreError: 99078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3016.847529] Lustre: server umount lustre-MDT0001 complete [ 3023.992640] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3025.945858] Key type lgssc unregistered [ 3026.166483] LNet: 102648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3026.175291] LNetError: 102648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3027.239313] LNet: Removed LNI 192.168.204.119@tcp [ 3027.799212] Key type .llcrypt unregistered [ 3027.802833] Key type ._llcrypt unregistered [ 3038.778874] Key type ._llcrypt registered [ 3038.781896] Key type .llcrypt registered [ 3038.871353] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3039.767462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.781292] alg: No test for adler32 (adler32-zlib) [ 3040.786755] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3040.986486] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3042.640218] Key type lgssc registered [ 3043.575655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.184445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.191413] LustreError: 103698:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3048.220278] LustreError: 103705:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3048.224942] LustreError: 103698:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3048.231795] LustreError: 103698:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3048.238119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.243873] LustreError: 103698:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3049.604457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.770089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.973181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.892600] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3062.132846] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing lsmod [ 3064.584539] LustreError: 104901:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3064.620327] LustreError: 104911:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3064.625942] LustreError: 104901:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4967 [ 3064.630885] LustreError: 104901:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3064.638271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.640104] LustreError: 104901:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3064.934318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.158954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.165950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.174662] Lustre: Skipped 1 previous similar message [ 3072.492236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.498727] Lustre: Skipped 4 previous similar messages [ 3073.413128] Lustre: server umount lustre-MDT0000 complete [ 3076.368343] LustreError: 103722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850054 with bad export cookie 8166933151571813420 [ 3076.380994] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3076.381899] LustreError: 103722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3076.651674] Lustre: server umount lustre-MDT0001 complete [ 3090.095348] Lustre: server umount lustre-OST0000 complete [ 3096.618563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.949852] LustreError: 105975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.968691] LustreError: 105975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3097.020567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.964674] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3102.182097] LustreError: 105976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.204753] LustreError: 105976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3106.503567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.768085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.887733] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3114.885676] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3118.607633] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3122.798641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.031520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.086646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3127.426437] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3131.808872] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3137.509332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.513031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3137.520653] Lustre: Skipped 2 previous similar messages [ 3137.527804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.809545] Lustre: server umount lustre-OST0000 complete [ 3147.746890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.749235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.753223] Lustre: Skipped 1 previous similar message [ 3147.757564] Lustre: Skipped 1 previous similar message [ 3147.760749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.441610] Lustre: server umount lustre-MDT0000 complete [ 3152.866189] LustreError: 105976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.887196] LustreError: 105976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3153.287335] LustreError: 105957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850131 with bad export cookie 8166933151571814099 [ 3153.291304] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3153.295622] LustreError: 105957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3153.562654] Lustre: server umount lustre-MDT0001 complete [ 3160.494454] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3162.400493] Key type lgssc unregistered [ 3162.640805] LNet: 109514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.650169] LNetError: 109514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.664293] LNet: Removed LNI 192.168.204.119@tcp [ 3163.310794] Key type .llcrypt unregistered [ 3163.317872] Key type ._llcrypt unregistered [ 3178.589832] Key type ._llcrypt registered [ 3178.593193] Key type .llcrypt registered [ 3178.682725] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3179.481279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3179.532764] alg: No test for adler32 (adler32-zlib) [ 3180.470104] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3180.643678] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3182.305232] Key type lgssc registered [ 3183.076047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3187.973605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3187.989092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.267577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.261401] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3198.216933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.498649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.516314] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3206.034331] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3209.607730] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3213.760556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.982633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.478254] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3221.923888] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3233.456078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3236.402812] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3238.423109] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3244.007879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.012068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.014249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.022990] Lustre: Skipped 1 previous similar message [ 3244.041041] Lustre: Skipped 1 previous similar message [ 3246.858800] Lustre: server umount lustre-OST0000 complete [ 3249.122581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.126196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.133788] Lustre: Skipped 1 previous similar message [ 3254.242678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.253417] Lustre: Skipped 2 previous similar messages [ 3255.370267] Lustre: server umount lustre-MDT0000 complete [ 3257.563265] LustreError: 110887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850236 with bad export cookie 4482056615595251835 [ 3257.565822] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.571682] LustreError: 110887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3257.766987] Lustre: server umount lustre-MDT0001 complete [ 3263.853209] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3265.411273] Key type lgssc unregistered [ 3265.608287] LNet: 114563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.613260] LNetError: 114563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.621577] LNet: Removed LNI 192.168.204.119@tcp [ 3266.142174] Key type .llcrypt unregistered [ 3266.144184] Key type ._llcrypt unregistered [ 3276.366806] Key type ._llcrypt registered [ 3276.371905] Key type .llcrypt registered [ 3276.437293] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3276.980160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.001037] alg: No test for adler32 (adler32-zlib) [ 3277.934145] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3278.062420] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3279.688449] Key type lgssc registered [ 3280.373606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3285.511299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3285.520772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.849789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.443349] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3294.639184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.875260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3297.396659] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3301.578208] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3304.761608] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3308.359809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.549344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.668242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3312.028176] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3316.421936] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3326.946044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3326.952211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.961712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.020730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3330.025970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.029611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.042316] Lustre: Skipped 1 previous similar message [ 3335.139043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.142800] Lustre: Skipped 1 previous similar message [ 3340.257614] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3340.260435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.268024] Lustre: Skipped 1 previous similar message [ 3340.354491] Lustre: server umount lustre-OST0000 complete [ 3345.378522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.379114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.384229] Lustre: Skipped 1 previous similar message [ 3348.792588] Lustre: server umount lustre-MDT0000 complete [ 3350.498183] LustreError: 115758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.511028] LustreError: 115758:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3350.940799] LustreError: 115738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850329 with bad export cookie 15705709897024702866 [ 3350.947954] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.139864] Lustre: server umount lustre-MDT0001 complete [ 3357.991505] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3359.720567] Key type lgssc unregistered [ 3359.942101] LNet: 119289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.948498] LNetError: 119289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3359.960078] LNet: Removed LNI 192.168.204.119@tcp [ 3360.423917] Key type .llcrypt unregistered [ 3360.425663] Key type ._llcrypt unregistered [ 3375.244541] Key type ._llcrypt registered [ 3375.246935] Key type .llcrypt registered [ 3375.329508] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3375.906711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.022879] alg: No test for adler32 (adler32-zlib) [ 3376.999299] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3377.198866] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3378.824314] Key type lgssc registered [ 3379.586896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3384.940936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3384.952462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.238387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.896571] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3394.073382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.342365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.082094] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3401.552388] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3404.838947] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3408.832251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.054829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.141724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3412.310546] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3416.566170] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3425.581709] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3427.740805] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3427.747496] Lustre: Skipped 2 previous similar messages [ 3430.439453] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3430.444298] Lustre: Skipped 2 previous similar messages [ 3432.483506] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3432.502789] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3437.844995] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3437.853850] Lustre: Skipped 2 previous similar messages [ 3437.914163] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.19@tcp 192.168.0.[2,10]@tcp [ 3444.243835] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.19@tcp 192.168.0.[2,10]@tcp [ 3445.940634] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.19/24@tcp in log lustre-MDT0000 [ 3445.945236] Lustre: Skipped 2 previous similar messages [ 3453.484041] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.19/24@tcp [ 3458.529756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.534831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.544119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.583050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.585869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.594707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.235662] Lustre: server umount lustre-OST0000 complete [ 3465.697461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.698188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.705027] Lustre: Skipped 1 previous similar message [ 3465.714458] Lustre: Skipped 2 previous similar messages [ 3470.818267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.826706] Lustre: Skipped 1 previous similar message [ 3471.882485] Lustre: server umount lustre-MDT0000 complete [ 3474.110733] LustreError: 120660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850452 with bad export cookie 16062358313160713462 [ 3474.114737] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.117239] LustreError: 120660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3474.321953] Lustre: server umount lustre-MDT0001 complete [ 3481.637584] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3483.249425] Key type lgssc unregistered [ 3483.434516] LNet: 125510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3483.439276] LNetError: 125510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3483.456822] LNet: Removed LNI 192.168.204.119@tcp [ 3484.013885] Key type .llcrypt unregistered [ 3484.015424] Key type ._llcrypt unregistered [ 3500.886551] Key type ._llcrypt registered [ 3500.888882] Key type .llcrypt registered [ 3500.957682] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3501.569335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3501.582250] alg: No test for adler32 (adler32-zlib) [ 3502.557274] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3502.722678] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3504.328898] Key type lgssc registered [ 3504.957704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.300189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.312840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.564023] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3510.570716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.583877] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.19/24@tcp [ 3512.904123] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3517.712067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.909364] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3517.928692] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.19/24@tcp [ 3517.956710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.648683] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3524.849617] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3528.207645] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3531.872710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.081043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.809954] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3536.187654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3540.038166] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3551.713288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.717221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.718143] LustreError: Skipped 1 previous similar message [ 3551.730699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.768022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.770217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.770728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.789203] Lustre: Skipped 1 previous similar message [ 3555.630648] Lustre: server umount lustre-OST0000 complete [ 3558.884094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3558.884520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.889822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.902781] Lustre: Skipped 1 previous similar message [ 3563.911572] Lustre: server umount lustre-MDT0000 complete [ 3564.004302] LustreError: 127047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.014191] LustreError: 127047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3565.987456] LustreError: 127025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850544 with bad export cookie 101350172882461435 [ 3565.990899] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3565.992707] LustreError: 127025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3566.170962] Lustre: server umount lustre-MDT0001 complete [ 3571.311307] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3572.851286] Key type lgssc unregistered [ 3573.019595] LNet: 130631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3573.025250] LNetError: 130631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3573.034767] LNet: Removed LNI 192.168.204.119@tcp [ 3573.564810] Key type .llcrypt unregistered [ 3573.567296] Key type ._llcrypt unregistered [ 3585.370282] Key type ._llcrypt registered [ 3585.371500] Key type .llcrypt registered [ 3585.417845] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 3593.235253] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3593.739414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3593.750547] alg: No test for adler32 (adler32-zlib) [ 3594.640273] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3594.745191] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3596.336224] Key type lgssc registered [ 3596.823824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3613.960545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3613.979561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.140194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3615.168522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3615.221723] Lustre: lustre-MDT0000: new disk, initializing [ 3615.275068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.287887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3617.494352] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3623.840275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.920509] Lustre: 134000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.964331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.969602] Lustre: Skipped 1 previous similar message [ 3624.015231] Lustre: lustre-MDT0001: new disk, initializing [ 3624.073561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3624.096290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3624.102631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3626.711508] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3632.256478] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3635.211498] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3638.581982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.741805] Lustre: lustre-OST0000: new disk, initializing [ 3638.744736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3638.748678] Lustre: 135966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3638.806954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.074913] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3647.448945] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3649.049052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3649.053841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3649.128534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.709536] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3649.830439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3652.142597] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3652.240046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3654.116562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3654.116980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.118711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3654.120458] LustreError: Skipped 1 previous similar message [ 3654.134890] Lustre: Skipped 1 previous similar message [ 3655.415519] Lustre: server umount lustre-OST0000 complete [ 3659.233948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.236297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.242795] Lustre: Skipped 1 previous similar message [ 3659.248538] Lustre: Skipped 2 previous similar messages [ 3663.481103] Lustre: server umount lustre-MDT0000 complete [ 3665.381553] LustreError: 133992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850644 with bad export cookie 13474106350477348134 [ 3665.384528] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.392601] LustreError: 133992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3665.572702] Lustre: server umount lustre-MDT0001 complete [ 3670.170771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.420611] LustreError: 138180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.429406] LustreError: 138180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3670.475083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.705550] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3675.618831] LustreError: 138181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.652050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.984313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3683.228443] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3695.809536] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3707.282780] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3723.229700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.186682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.329992] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3747.486457] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.310600] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3755.565925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3761.252298] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3761.538273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3770.049050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.232449] Lustre: lustre-OST0001: new disk, initializing [ 3770.236578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3770.242541] Lustre: 141557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3770.353434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3776.218942] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3776.594066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3776.600755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3776.682672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3784.421836] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3784.715086] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3789.580654] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3789.833298] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3796.555604] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97c989d9e800.ost_server_uuid 50 [ 3798.395429] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97c989d9e800.ost_server_uuid in FULL state after 0 sec [ 3807.714678] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.723309] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.731151] LustreError: Skipped 1 previous similar message [ 3807.736297] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3807.746494] Lustre: Skipped 1 previous similar message [ 3809.297341] Lustre: server umount lustre-OST0001 complete [ 3812.842041] LustreError: 141940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.868683] LustreError: 141940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3816.418155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.427672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.448768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.454231] Lustre: Skipped 1 previous similar message [ 3817.977805] LustreError: 141940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.997818] LustreError: 141940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3819.626839] Lustre: server umount lustre-OST0000 complete [ 3827.687756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3827.704961] LustreError: Skipped 1 previous similar message [ 3827.718408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.730027] Lustre: Skipped 1 previous similar message [ 3827.735399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.741601] Lustre: Skipped 2 previous similar messages [ 3830.101558] Lustre: server umount lustre-MDT0000 complete [ 3833.313699] LustreError: 138181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.326417] LustreError: 138181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3833.647628] LustreError: 138163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850812 with bad export cookie 13474106350477349016 [ 3833.656118] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.658488] LustreError: 138163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3833.892488] Lustre: server umount lustre-MDT0001 complete [ 3859.387646] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3863.181556] Key type lgssc unregistered [ 3863.655600] LNet: 144559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.666549] LNetError: 144559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.680913] LNet: Removed LNI 192.168.204.119@tcp [ 3864.989315] Key type .llcrypt unregistered [ 3864.993190] Key type ._llcrypt unregistered [ 3907.922844] Key type ._llcrypt registered [ 3907.928204] Key type .llcrypt registered [ 3908.036955] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 3921.779115] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3922.701625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.726595] alg: No test for adler32 (adler32-zlib) [ 3923.828675] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 3924.103615] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3925.792265] Key type lgssc registered [ 3926.765381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3967.557091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3967.586566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.956198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3968.986918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3969.120935] Lustre: lustre-MDT0000: new disk, initializing [ 3969.224341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.248635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3974.562963] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3986.230542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.449584] Lustre: 148726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3986.518633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3986.522304] Lustre: Skipped 1 previous similar message [ 3986.580473] Lustre: lustre-MDT0001: new disk, initializing [ 3986.680725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.737881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3986.754463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3992.385161] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4003.111627] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4010.054946] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.224741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.531712] Lustre: lustre-OST0000: new disk, initializing [ 4018.535830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4018.544261] Lustre: 150695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4018.644057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.837450] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4028.473869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4028.480654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4028.568883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4035.997912] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4040.323234] Lustre: Failing over lustre-OST0000 [ 4040.648950] Lustre: server umount lustre-OST0000 complete [ 4057.575516] Lustre: 146945:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851020/real 1788851020] req@000000007fa8343f x1875746123449728/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788851036 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4057.592927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.604603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.619072] Lustre: 146946:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851022/real 1788851022] req@00000000caf9f622 x1875746123450624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788851038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4059.649108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.898244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.915745] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4060.960262] Lustre: 146947:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851022/real 1788851022] req@00000000f9eeae89 x1875746123451008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788851038 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4060.984927] Lustre: 146947:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4065.953229] Lustre: 146948:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851028/real 1788851028] req@0000000056511b20 x1875746123452032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788851044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4066.000138] Lustre: 146948:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4066.426067] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4068.073982] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4070.234410] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4070.234473] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4070.493786] Lustre: Failing over lustre-MDT0000 [ 4070.887812] Lustre: server umount lustre-MDT0000 complete [ 4071.265307] Lustre: 146945:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851033/real 1788851033] req@00000000a8edac19 x1875746123453184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788851049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4073.187607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.215829] Lustre: Skipped 2 previous similar messages [ 4078.564360] LustreError: 149890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.583132] LustreError: 149890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4079.917890] LustreError: 148731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.683781] LustreError: 148736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.702788] LustreError: 148736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4088.736130] Lustre: 146946:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851051/real 1788851051] req@000000003791bd85 x1875746123465088/t0(0) o400->MGC192.168.204.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788851067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4088.770949] Lustre: 146946:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4088.786175] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4088.799798] LustreError: 148737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.817957] LustreError: 148737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4090.503821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.923879] LustreError: 148732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.972831] LustreError: 148732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4099.040763] LustreError: 146944:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f7ad7a38 x1875746123469824/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4099.405253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.498653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4100.402036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4104.555592] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4104.715942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4104.732127] Lustre: Skipped 1 previous similar message [ 4104.797689] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4104.908732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4116.453045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.461732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.478372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.050601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.056331] Lustre: Skipped 1 previous similar message [ 4121.915193] Lustre: server umount lustre-OST0000 complete [ 4130.291525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.308494] Lustre: Skipped 1 previous similar message [ 4130.316072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4130.325224] Lustre: Skipped 1 previous similar message [ 4132.788530] Lustre: server umount lustre-MDT0000 complete [ 4135.394779] LustreError: 149890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.442500] LustreError: 149890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4137.256739] LustreError: 150694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851115 with bad export cookie 258669635572535337 [ 4137.258525] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4137.267798] LustreError: 150694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4137.695058] Lustre: server umount lustre-MDT0001 complete [ 4150.266260] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4153.892486] Key type lgssc unregistered [ 4154.311599] LNet: 154351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4154.325630] LNetError: 154351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4154.365245] LNet: Removed LNI 192.168.204.119@tcp [ 4155.894921] Key type .llcrypt unregistered [ 4155.898226] Key type ._llcrypt unregistered [ 4199.547139] Key type ._llcrypt registered [ 4199.549273] Key type .llcrypt registered [ 4199.678335] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4200.845762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4200.863649] alg: No test for adler32 (adler32-zlib) [ 4202.000209] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 4202.259288] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4203.992289] Key type lgssc registered [ 4205.285577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4214.502426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4214.520626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.072034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.639278] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4225.510163] LustreError: 155872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.529692] LustreError: 155872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4230.626853] LustreError: 155873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.713905] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4234.285046] LustreError: 155901:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4234.294293] LustreError: 155901:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4234.302425] LustreError: 155901:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4234.681500] Lustre: server umount lustre-MDT0000 complete [ 4249.874719] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 4259.730550] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4304.289965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.476885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4304.509317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4304.658789] Lustre: lustre-MDT0000: new disk, initializing [ 4304.810073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.839471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4309.950833] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4321.409123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.565924] Lustre: 159212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4321.642672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4321.653311] Lustre: Skipped 1 previous similar message [ 4321.760744] Lustre: lustre-MDT0001: new disk, initializing [ 4321.853187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4321.889076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4321.908934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4326.391172] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4336.363853] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4342.760282] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4349.643109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.065555] Lustre: lustre-OST0000: new disk, initializing [ 4350.070131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4350.076365] Lustre: 161179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4350.170469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.203371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4357.218820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4357.367500] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4357.408971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.093590] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.113591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4374.326984] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:09:10 (1788851350) [ 4654.561407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.562714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.580490] Lustre: Skipped 1 previous similar message [ 4654.594644] Lustre: Skipped 2 previous similar messages [ 4659.682430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4659.692149] Lustre: Skipped 2 previous similar messages [ 4660.778977] Lustre: server umount lustre-MDT0000 complete [ 4664.801372] LustreError: 159223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.844164] LustreError: 159223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4665.426602] LustreError: 159202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851644 with bad export cookie 10351280979618984464 [ 4665.427936] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.439072] LustreError: 159202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4669.921164] LustreError: 160286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.921933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.958818] Lustre: Skipped 1 previous similar message [ 4669.964743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4671.883714] Lustre: server umount lustre-MDT0001 complete [ 4690.400243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4690.645888] Lustre: server umount lustre-OST0000 complete [ 4701.976935] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 4714.945890] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4752.657886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.932726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4752.955974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4753.034298] Lustre: lustre-MDT0000: new disk, initializing [ 4753.118508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.133297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4757.836802] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4770.267696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.388545] Lustre: 165847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4770.439027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4770.441938] Lustre: Skipped 1 previous similar message [ 4770.499437] Lustre: lustre-MDT0001: new disk, initializing [ 4770.574697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4770.629631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4770.646568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4775.301482] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4785.621434] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4792.780700] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4800.437151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.705884] Lustre: lustre-OST0000: new disk, initializing [ 4800.709719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4800.717881] Lustre: 167818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4800.821780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.725333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4806.738648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4806.835911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4807.131363] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4817.085828] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4821.653672] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4821.882326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4826.729587] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4826.926380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4832.744555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.755401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4832.760612] Lustre: Skipped 1 previous similar message [ 4832.761492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4832.792196] Lustre: Skipped 2 previous similar messages [ 4835.411806] Lustre: server umount lustre-OST0000 complete [ 4842.470136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4842.479788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.489249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4845.975443] Lustre: server umount lustre-MDT0000 complete [ 4848.098523] LustreError: 168538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.124466] LustreError: 168538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4850.308753] LustreError: 165842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851828 with bad export cookie 10351280979619143287 [ 4850.313566] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.316541] LustreError: 165842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4850.704763] Lustre: server umount lustre-MDT0001 complete [ 4859.457893] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:17:16 (1788851836) [ 4879.279398] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 4893.198629] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4936.121279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.329777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4936.359300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4936.498855] Lustre: lustre-MDT0000: new disk, initializing [ 4936.607877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.629971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4942.286856] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4956.367384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.562718] Lustre: 172574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4956.888825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4962.036788] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4971.969983] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4978.065798] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4984.730846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4985.026288] Lustre: lustre-OST0000: new disk, initializing [ 4985.029558] Lustre: Skipped 1 previous similar message [ 4985.044913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4985.050357] Lustre: Skipped 2 previous similar messages [ 4985.059287] Lustre: 174555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4991.221449] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4992.603799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4992.620432] Lustre: Skipped 1 previous similar message [ 4992.637199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4992.761555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5001.376287] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5008.364853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.380769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.385129] Lustre: Skipped 1 previous similar message [ 5008.390117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.400199] LustreError: Skipped 1 previous similar message [ 5008.408319] Lustre: Skipped 2 previous similar messages [ 5022.176958] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5022.476471] Lustre: server umount lustre-OST0000 complete [ 5028.324175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5028.337292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.361485] Lustre: Skipped 1 previous similar message [ 5028.388978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5028.398173] Lustre: Skipped 4 previous similar messages [ 5032.887911] Lustre: server umount lustre-MDT0000 complete [ 5033.964783] LustreError: 172586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.996695] LustreError: 172586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5037.158490] LustreError: 172568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852015 with bad export cookie 10351280979619144176 [ 5037.161155] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.180037] LustreError: 172568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5037.711907] Lustre: server umount lustre-MDT0001 complete [ 5048.325583] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:20:24 (1788852024) [ 5070.265840] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 5086.379728] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5130.102645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.320863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5130.346498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5130.415712] Lustre: lustre-MDT0000: new disk, initializing [ 5130.490445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.497358] Lustre: Skipped 2 previous similar messages [ 5130.514952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5135.383624] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5146.338260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.487283] Lustre: 178916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5146.706557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5150.811771] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5159.681882] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5165.935321] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5171.945308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5172.171778] Lustre: 180896:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5177.845758] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5179.436130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5179.608255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5188.549649] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5195.246640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.246727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.259211] Lustre: Skipped 1 previous similar message [ 5195.261109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.280702] Lustre: Skipped 2 previous similar messages [ 5201.390745] Lustre: server umount lustre-OST0000 complete [ 5211.724113] Lustre: server umount lustre-MDT0000 complete [ 5215.736769] LustreError: 178930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.758688] LustreError: 178930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5215.800703] LustreError: 178910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852194 with bad export cookie 10351280979619145261 [ 5215.803263] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5215.814916] LustreError: 178910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5216.192852] Lustre: server umount lustre-MDT0001 complete [ 5229.346352] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5232.652756] Key type lgssc unregistered [ 5233.077967] LNet: 182751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5233.084386] LNetError: 182751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.105954] LNet: Removed LNI 192.168.204.119@tcp [ 5234.166114] Key type .llcrypt unregistered [ 5234.167781] Key type ._llcrypt unregistered [ 5259.991714] Key type ._llcrypt registered [ 5259.994340] Key type .llcrypt registered [ 5260.132679] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5261.366885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.411522] alg: No test for adler32 (adler32-zlib) [ 5262.476927] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 5262.688587] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5264.393070] Key type lgssc registered [ 5265.788311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5273.858145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5273.901710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.684676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.518863] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5285.861085] LustreError: 184164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.887643] LustreError: 184164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5290.026256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.357994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5295.304923] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5303.797943] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5311.416242] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5322.493467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5323.284708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.100785] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5341.376185] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5345.862120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5365.221244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5365.227205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.236442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5365.250155] Lustre: Skipped 1 previous similar message [ 5370.341956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5370.366944] Lustre: Skipped 2 previous similar messages [ 5371.586791] Lustre: server umount lustre-OST0000 complete [ 5377.509262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5377.518873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.532407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5380.579225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5380.579303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.693358] Lustre: server umount lustre-MDT0000 complete [ 5385.697883] LustreError: 184165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.744265] LustreError: 184165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5387.401074] LustreError: 184146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852366 with bad export cookie 15534169893382723077 [ 5387.413165] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.420891] LustreError: 184146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5387.836801] Lustre: server umount lustre-MDT0001 complete [ 5400.008350] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5403.229254] Key type lgssc unregistered [ 5403.568817] LNet: 187723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5403.584426] LNetError: 187723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5404.652165] LNet: Removed LNI 192.168.204.119@tcp [ 5405.723550] Key type .llcrypt unregistered [ 5405.727865] Key type ._llcrypt unregistered [ 5432.916808] Key type ._llcrypt registered [ 5432.919072] Key type .llcrypt registered [ 5433.070813] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5433.883941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5433.915120] alg: No test for adler32 (adler32-zlib) [ 5435.137960] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 5435.442753] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5437.160965] Key type lgssc registered [ 5438.601734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5446.930354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5446.949784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.418790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5452.946825] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5458.924125] LustreError: 189137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.966594] LustreError: 189137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5463.363642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.932949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5468.987746] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5478.189592] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5486.022703] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5495.588731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5495.851327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.223597] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5511.064789] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5519.410205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5529.570935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.596194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.599637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.628826] Lustre: Skipped 1 previous similar message [ 5529.667809] Lustre: Skipped 1 previous similar message [ 5532.003905] Lustre: lustre-OST0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 5534.692518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5534.710461] Lustre: Skipped 1 previous similar message [ 5537.076260] Lustre: lustre-OST0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 5541.344537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.704080] Lustre: server umount lustre-OST0000 complete [ 5556.672182] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97c9880a9800.ost_server_uuid 50 [ 5559.413363] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5574.626225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.653775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.665118] Lustre: Skipped 3 previous similar messages [ 5579.673698] Lustre: server umount lustre-MDT0000 complete [ 5579.749269] LustreError: 189132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.765312] LustreError: 189132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5583.986146] LustreError: 189118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852562 with bad export cookie 11329257023585998891 [ 5583.986764] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.995620] LustreError: 189118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5584.343533] Lustre: server umount lustre-MDT0001 complete [ 5593.985878] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:29:30 (1788852570) [ 5601.310044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.857664] LustreError: 193082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.982923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.397360] LustreError: 193083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.893501] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5611.493627] LustreError: 193082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.236355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.741745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5625.893672] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5634.697242] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5641.876386] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5649.909221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.278728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.030995] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5665.435985] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5671.812059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5673.430729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.732857] Lustre: lustre-OST0001: new disk, initializing [ 5673.741134] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5673.751311] Lustre: 195925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5673.854054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5681.804752] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5682.845412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5682.860052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5683.028064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5693.784911] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5697.508334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.523098] LustreError: Skipped 1 previous similar message [ 5697.526557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.565121] Lustre: Skipped 1 previous similar message [ 5697.604150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.668939] LustreError: 196325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.702176] LustreError: 196325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5704.057183] Lustre: server umount lustre-OST0000 complete [ 5713.472778] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5713.792983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5719.881920] LustreError: 194905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.902323] LustreError: 194905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5720.282767] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5720.644598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5739.509737] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.522137] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.544712] LustreError: Skipped 1 previous similar message [ 5739.550516] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5739.550525] Lustre: Skipped 3 previous similar messages [ 5739.632930] Lustre: Skipped 2 previous similar messages [ 5743.081660] Lustre: server umount lustre-OST0001 complete [ 5749.693665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.704943] Lustre: Skipped 1 previous similar message [ 5756.260185] Lustre: server umount lustre-MDT0000 complete [ 5759.987183] LustreError: 194011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.003456] LustreError: 194011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5760.927391] LustreError: 193063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852739 with bad export cookie 11329257023585999962 [ 5760.936857] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.948274] LustreError: 193063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5761.729402] Lustre: server umount lustre-MDT0001 complete [ 5790.831774] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:32:47 (1788852767) [ 5798.187927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.431840] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5798.468474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5798.748088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.308953] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5814.844986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.976386] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5815.027915] Lustre: 199369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5821.088624] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5830.647262] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5837.339455] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5845.872925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5846.101333] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5853.795232] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5855.940744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5863.500659] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5871.693461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5871.802424] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5872.151703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5872.161682] Lustre: Skipped 2 previous similar messages [ 5878.532053] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5887.627212] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.535939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.540128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.540137] Lustre: Skipped 5 previous similar messages [ 5890.581315] Lustre: Skipped 1 previous similar message [ 5896.702629] Lustre: server umount lustre-OST0000 complete [ 5898.544100] LustreError: 201218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.134854] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.135512] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.172082] Lustre: Skipped 1 previous similar message [ 5919.524536] Lustre: server umount lustre-OST0001 complete [ 5938.144749] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5938.528157] Lustre: server umount lustre-MDT0000 complete [ 5942.947778] LustreError: 199362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852921 with bad export cookie 11329257023586001369 [ 5942.948446] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5942.971902] LustreError: 199362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5943.477949] Lustre: server umount lustre-MDT0001 complete [ 5974.136436] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:35:50 (1788852950) [ 5999.675734] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 6012.526049] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 6058.506330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.752440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6058.798439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6058.896542] Lustre: lustre-MDT0000: new disk, initializing [ 6058.974845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.001906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6064.030311] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6076.337353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.545617] Lustre: 207478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6076.652310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6076.656886] Lustre: Skipped 1 previous similar message [ 6076.758519] Lustre: lustre-MDT0001: new disk, initializing [ 6077.046644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6077.055229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6083.126236] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6095.118184] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6103.403709] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6112.082356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.568642] Lustre: lustre-OST0000: new disk, initializing [ 6112.572523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6112.582814] Lustre: 209448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6118.897980] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6119.550845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6119.568522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6119.680748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6129.834397] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6135.163511] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6135.364339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6139.884379] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6140.163978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6145.509279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.525762] Lustre: Skipped 2 previous similar messages [ 6145.530308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.533498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.533676] Lustre: Skipped 12 previous similar messages [ 6145.543853] LustreError: Skipped 1 previous similar message [ 6148.494322] Lustre: server umount lustre-OST0000 complete [ 6153.707694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6159.061421] Lustre: server umount lustre-MDT0000 complete [ 6160.867975] LustreError: 207489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.889557] LustreError: 207489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6163.363047] LustreError: 207469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853141 with bad export cookie 11329257023586002930 [ 6163.370431] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.377672] LustreError: 207469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.827647] Lustre: server umount lustre-MDT0001 complete [ 6174.283646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.969321] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6194.395368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.078360] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6210.562746] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6217.951721] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6226.193655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.700947] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6245.124418] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6251.675612] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6251.926347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6258.142688] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6258.368685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6265.329527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.343639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.352879] LustreError: Skipped 1 previous similar message [ 6265.400922] Lustre: Skipped 4 previous similar messages [ 6267.638222] Lustre: server umount lustre-OST0000 complete [ 6276.333402] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6282.101568] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6312.138548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.548646] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6328.695759] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.596434] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6330.923174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.951234] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.955217] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6330.993431] Lustre: Skipped 1 previous similar message [ 6335.971156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6335.973606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6335.984089] Lustre: Skipped 7 previous similar messages [ 6349.280291] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6349.479709] Lustre: server umount lustre-OST0000 complete [ 6360.398496] Lustre: server umount lustre-MDT0000 complete [ 6364.719326] LustreError: 212400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853343 with bad export cookie 11329257023586003812 [ 6364.724927] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6364.739763] LustreError: 212400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6365.157220] Lustre: server umount lustre-MDT0001 complete [ 6376.486737] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:42:33 (1788853353) [ 6383.905542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.773472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.792937] Lustre: Skipped 6 previous similar messages [ 6392.964394] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6408.122099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.725519] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6421.495050] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6428.289732] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6435.949787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.600254] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6455.724151] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6461.700042] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6461.999832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6468.123328] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6468.411237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6477.819275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6478.175480] Lustre: lustre-OST0001: new disk, initializing [ 6478.182428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6478.192237] Lustre: 221012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6486.704731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6486.744572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6486.906671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6487.142345] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6498.657683] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6504.570454] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6504.766229] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6511.161626] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6511.501099] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6517.736339] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.737336] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6517.756471] Lustre: Skipped 5 previous similar messages [ 6520.459076] Lustre: server umount lustre-OST0001 complete [ 6522.865263] LustreError: 219460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.878263] LustreError: 219460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6527.621951] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6527.827723] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6533.874207] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6534.267164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6565.902148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.510683] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6571.536764] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6571.547586] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6572.787588] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6581.629323] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6583.548182] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6591.978298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6591.987495] Lustre: Skipped 13 previous similar messages [ 6592.343476] Lustre: server umount lustre-OST0001 complete [ 6625.248538] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6625.517202] Lustre: server umount lustre-OST0000 complete [ 6637.495504] Lustre: server umount lustre-MDT0000 complete [ 6642.928927] LustreError: 217619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853621 with bad export cookie 11329257023586004778 [ 6642.936913] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6642.937957] LustreError: 217619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6649.486316] Lustre: server umount lustre-MDT0001 complete [ 6677.139512] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:47:34 (1788853654) [ 6686.973543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.493199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6687.594393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6694.445718] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6709.318271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.545580] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6709.598239] Lustre: 226259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6716.354560] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6727.190880] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6736.873529] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.175041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.933701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6762.279630] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6772.186941] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.427083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.566090] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6791.367180] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6804.526660] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6806.912230] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6813.528526] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6813.832977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6819.751233] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6820.101301] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6828.143552] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97c98660e000.ost_server_uuid 50 [ 6830.116505] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97c98660e000.ost_server_uuid in IDLE state after 0 sec [ 6831.484315] Lustre: Permanently deactivating lustre-OST0001 [ 6831.497478] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6836.244833] Lustre: Permanently reactivating lustre-OST0001 [ 6836.298360] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6836.310481] Lustre: Skipped 2 previous similar messages [ 6841.321156] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6841.340846] LustreError: Skipped 3 previous similar messages [ 6841.344216] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.361297] Lustre: Skipped 6 previous similar messages [ 6847.470470] Lustre: server umount lustre-OST0001 complete [ 6858.529876] Lustre: server umount lustre-OST0000 complete [ 6869.825445] Lustre: server umount lustre-MDT0000 complete [ 6874.674062] LustreError: 226251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853853 with bad export cookie 11329257023586006066 [ 6874.685810] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6874.707710] LustreError: 226251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6875.457590] Lustre: server umount lustre-MDT0001 complete [ 6904.287526] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:51:20 (1788853880) [ 6913.249451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.682767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6913.721630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6913.734110] Lustre: Skipped 2 previous similar messages [ 6913.927228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.938557] Lustre: Skipped 8 previous similar messages [ 6919.403184] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6929.652227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.815974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6929.875575] Lustre: 232963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6935.451980] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6944.312864] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6951.649828] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.622037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.858567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6959.883899] Lustre: Permanently deactivating lustre-OST0000 [ 6959.897306] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6967.371448] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 6967.401389] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6967.429052] LustreError: 235190:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6967.449596] LustreError: 235190:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6975.871652] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6983.123413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6983.202561] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6991.072437] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6991.502773] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7002.673491] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7005.686887] Lustre: Permanently reactivating lustre-OST0000 [ 7005.729203] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7005.742430] Lustre: Skipped 2 previous similar messages [ 7015.674597] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7015.676787] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 00000000ef2452a7, cur 1788853994 deadline 1788853976 last 1788853946 [ 7015.709840] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7015.765884] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7015.779941] Lustre: Skipped 1 previous similar message [ 7015.855974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7029.826673] Lustre: server umount lustre-OST0001 complete [ 7037.470621] LustreError: 236171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.517845] LustreError: 236171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 47 previous similar messages [ 7049.185287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7049.539406] Lustre: server umount lustre-OST0000 complete [ 7063.075603] Lustre: server umount lustre-MDT0000 complete [ 7068.162376] LustreError: 233705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854046 with bad export cookie 11329257023586007711 [ 7068.175364] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.181051] LustreError: 233705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7068.746304] Lustre: server umount lustre-MDT0001 complete [ 7083.176630] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 7086.243163] Key type lgssc unregistered [ 7086.513724] LNet: 237848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7086.526290] LNetError: 237848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7086.545163] LNet: Removed LNI 192.168.204.119@tcp [ 7087.531119] Key type .llcrypt unregistered [ 7087.535343] Key type ._llcrypt unregistered [ 7112.785611] Key type ._llcrypt registered [ 7112.792684] Key type .llcrypt registered [ 7113.026220] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7115.247427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7115.489240] alg: No test for adler32 (adler32-zlib) [ 7116.864251] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7117.179571] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 7118.976821] Key type lgssc registered [ 7121.719302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7138.637274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7138.687713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.778350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.890937] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7151.074666] LustreError: 239314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.089424] LustreError: 239314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7156.199039] LustreError: 239315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.620680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.823367] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7157.895453] Lustre: 239302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7157.923946] Lustre: Permanently deactivating lustre-MDT0001 [ 7157.995944] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7158.650764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7165.011871] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7166.027437] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7177.094381] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.805368] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7195.170158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7195.654396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.884900] LustreError: 241159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.096903] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7208.949123] LustreError: 241160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.999907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7213.103926] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7216.100957] LustreError: 241158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.138938] LustreError: 241158:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7220.890741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.162098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7226.363821] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7226.372874] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7228.241816] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7237.226062] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7241.036377] Lustre: Permanently reactivating lustre-MDT0001 [ 7241.078847] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7241.097874] Lustre: Skipped 2 previous similar messages [ 7246.681225] Lustre: lustre-MDT0001: Client fb4be465-a2b0-426f-998d-728b30c81e43 (at 192.168.204.19@tcp) reconnecting [ 7247.210151] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.228350] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7247.237923] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7247.251986] LustreError: 242886:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7247.259825] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7247.268134] LustreError: 242886:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7247.268302] LustreError: 242886:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.119@tcp: cfg command failed: rc = -108 [ 7247.268334] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7247.268334] [ 7247.269869] LustreError: 239306:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7255.861213] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7258.102461] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7264.830700] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7265.069706] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7270.308823] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7270.535469] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7274.127990] Lustre: Permanently deactivating lustre-MDT0001 [ 7274.145642] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7274.152203] Lustre: Skipped 2 previous similar messages [ 7283.414321] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7293.923697] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7293.925159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.934396] Lustre: Skipped 3 previous similar messages [ 7293.943824] Lustre: Skipped 3 previous similar messages [ 7299.066291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7299.292869] Lustre: server umount lustre-MDT0000 complete [ 7304.165297] LustreError: 240059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.185128] LustreError: 240059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7304.557870] LustreError: 239295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854283 with bad export cookie 15542547817786369026 [ 7304.590334] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7304.598600] LustreError: 239295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7305.059834] Lustre: server umount lustre-MDT0001 complete [ 7310.473758] Lustre: server umount lustre-OST0000 complete [ 7327.926753] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:58:24 (1788854304) [ 7341.579100] Lustre: server umount lustre-OST0001 complete [ 7351.008765] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 7363.474953] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7410.121938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.455192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7410.470057] Lustre: Skipped 2 previous similar messages [ 7410.521681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7410.599386] Lustre: lustre-MDT0000: new disk, initializing [ 7410.686863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7410.724968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7416.312275] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7429.470935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.587889] Lustre: 247605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7429.683314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7429.691028] Lustre: Skipped 1 previous similar message [ 7429.802650] Lustre: lustre-MDT0001: new disk, initializing [ 7429.911249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.981597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7429.992287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7435.342624] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7445.771179] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7451.962556] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7459.564937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7459.866826] Lustre: lustre-OST0000: new disk, initializing [ 7459.872633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7459.884316] Lustre: 249575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7459.970414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.046351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7466.065438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7466.271861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7467.012922] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7479.097095] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7483.723659] LustreError: 249372:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7493.792077] LustreError: 249372:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7493.889332] LustreError: 247612:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7495.381772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.695881] Lustre: lustre-OST0001: new disk, initializing [ 7495.707129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7495.717123] Lustre: 250701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7495.920890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7502.071944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7502.096758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7502.207022] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7503.964996] LustreError: 247612:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7505.317930] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7517.896214] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7519.592746] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7522.477714] Lustre: server umount lustre-OST0001 complete [ 7522.800384] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.803613] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7522.833306] LustreError: 249558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.837907] LustreError: Skipped 1 previous similar message [ 7522.860324] LustreError: 249558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7543.272482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7543.276928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.296392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7543.316447] Lustre: Skipped 2 previous similar messages [ 7543.329764] Lustre: Skipped 4 previous similar messages [ 7546.381996] Lustre: server umount lustre-OST0000 complete [ 7552.996373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7553.005240] LustreError: Skipped 1 previous similar message [ 7553.018906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.037790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.546965] Lustre: server umount lustre-MDT0000 complete [ 7558.628398] LustreError: 247617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.679193] LustreError: 247617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7560.574584] LustreError: 247597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854539 with bad export cookie 15542547817786371623 [ 7560.576133] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7560.595555] LustreError: 247597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7567.333849] Lustre: server umount lustre-MDT0001 complete [ 7580.143887] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 7583.077613] Key type lgssc unregistered [ 7583.411839] LNet: 252733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7583.431172] LNetError: 252733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7583.474247] LNet: Removed LNI 192.168.204.119@tcp [ 7584.463411] Key type .llcrypt unregistered [ 7584.467107] Key type ._llcrypt unregistered [ 7632.027205] Key type ._llcrypt registered [ 7632.032593] Key type .llcrypt registered [ 7632.202811] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7633.739091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7633.764357] alg: No test for adler32 (adler32-zlib) [ 7634.740596] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7634.984500] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 7636.736265] Key type lgssc registered [ 7638.107161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7646.878567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7646.894945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.234410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7648.286274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7648.503358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.516754] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7668.650511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.866033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7668.917824] Lustre: 254940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7669.633299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7676.741427] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7685.511563] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.983289] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7700.498253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.701772] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7700.898629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.803412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7708.591306] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7717.530649] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7778.273670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7778.283923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.297365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7778.790252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7778.803160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.823869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.122412] Lustre: server umount lustre-OST0000 complete [ 7789.025240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7807.234490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.698853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.044575] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7818.920903] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7818.940747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7818.944162] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7823.183390] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7865.313900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7865.323714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7865.330300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7865.348917] Lustre: Skipped 1 previous similar message [ 7869.491957] Lustre: server umount lustre-OST0000 complete [ 7874.529652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7874.544841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.572723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.580456] Lustre: Skipped 1 previous similar message [ 7879.806796] Lustre: server umount lustre-MDT0000 complete [ 7883.597772] LustreError: 254931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854862 with bad export cookie 14245004544514653259 [ 7883.606352] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.613662] LustreError: 254931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7884.058128] Lustre: server umount lustre-MDT0001 complete [ 7898.326845] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 7902.325690] Key type lgssc unregistered [ 7902.727622] LNet: 260367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7902.738741] LNetError: 260367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7902.761066] LNet: Removed LNI 192.168.204.119@tcp [ 7903.738116] Key type .llcrypt unregistered [ 7903.744174] Key type ._llcrypt unregistered [ 7933.862743] Key type ._llcrypt registered [ 7933.865526] Key type .llcrypt registered [ 7933.954463] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 7934.791912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7934.824532] alg: No test for adler32 (adler32-zlib) [ 7936.070604] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 7936.343886] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 7938.081685] Key type lgssc registered [ 7939.246600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7947.676615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7947.696455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.156248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.050471] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7958.503049] LustreError: 261787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.531077] LustreError: 261787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7963.617666] LustreError: 261788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7964.208141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.566589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7969.590574] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 7978.162450] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7984.994591] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7992.314373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.596703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.668354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7999.652693] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8008.284224] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8039.905953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8039.917465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.932449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8040.929694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8040.934346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8040.941124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.946895] Lustre: Skipped 1 previous similar message [ 8042.854679] Lustre: server umount lustre-OST0000 complete [ 8051.170232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.194946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.202643] Lustre: Skipped 1 previous similar message [ 8052.945042] Lustre: server umount lustre-MDT0000 complete [ 8056.291200] LustreError: 264159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.337236] LustreError: 264159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8056.747715] LustreError: 261769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855035 with bad export cookie 7855937791846815444 [ 8056.750535] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8056.763038] LustreError: 261769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8057.276824] Lustre: server umount lustre-MDT0001 complete [ 8069.027950] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8071.744968] Key type lgssc unregistered [ 8072.093249] LNet: 266096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8072.100169] LNetError: 266096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8073.128857] LNet: Removed LNI 192.168.204.119@tcp [ 8073.958807] Key type .llcrypt unregistered [ 8073.964615] Key type ._llcrypt unregistered [ 8092.110262] Key type ._llcrypt registered [ 8092.114290] Key type .llcrypt registered [ 8092.273724] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8093.275423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8093.298959] alg: No test for adler32 (adler32-zlib) [ 8094.451863] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 8094.732652] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 8096.488300] Key type lgssc registered [ 8097.527856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8106.378416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8106.423865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.038162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.112683] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8118.244084] LustreError: 267331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8118.269455] LustreError: 267331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8123.024117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.362366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8123.524505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8129.159575] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8141.439131] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8150.278743] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8159.806789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8160.108350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.900704] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8176.965608] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8177.524955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8189.921982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8189.945435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.976760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8192.998783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8193.005631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8193.019073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.885508] Lustre: server umount lustre-OST0000 complete [ 8203.235387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.236441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.253897] Lustre: Skipped 1 previous similar message [ 8203.271922] Lustre: Skipped 2 previous similar messages [ 8207.670966] Lustre: server umount lustre-MDT0000 complete [ 8208.354102] LustreError: 268572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8208.374347] LustreError: 268572:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8212.679294] LustreError: 267312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855191 with bad export cookie 4162192108419766931 [ 8212.684745] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8212.688452] LustreError: 267312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8213.249234] Lustre: server umount lustre-MDT0001 complete [ 8227.282528] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8231.294422] Key type lgssc unregistered [ 8231.599684] LNet: 271011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8231.623836] LNetError: 271011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8232.684745] LNet: Removed LNI 192.168.204.119@tcp [ 8234.221618] Key type .llcrypt unregistered [ 8234.227759] Key type ._llcrypt unregistered [ 8266.088351] Key type ._llcrypt registered [ 8266.091700] Key type .llcrypt registered [ 8266.251432] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8268.413553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8268.560997] alg: No test for adler32 (adler32-zlib) [ 8269.938227] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 8270.387414] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 8272.128238] Key type lgssc registered [ 8273.539540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8283.280452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8283.326378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.902748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8289.646958] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8295.398821] LustreError: 272425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.418745] LustreError: 272425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8298.911706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.441377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8303.487977] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8311.260421] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8318.304229] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8324.975505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8325.226934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.389683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8331.649853] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8341.548303] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8372.705322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8372.716796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.731923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.767491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.769979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.788966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.843633] Lustre: Skipped 1 previous similar message [ 8377.191517] Lustre: server umount lustre-OST0000 complete [ 8385.000799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.014879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8385.019728] Lustre: Skipped 1 previous similar message [ 8385.037976] Lustre: Skipped 1 previous similar message [ 8390.118545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8390.126944] Lustre: Skipped 1 previous similar message [ 8395.240177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8395.255099] Lustre: Skipped 1 previous similar message [ 8396.256299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8396.491278] Lustre: server umount lustre-MDT0000 complete [ 8400.354416] LustreError: 272422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.380103] LustreError: 272422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8401.144798] LustreError: 272406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855379 with bad export cookie 17553376227887581518 [ 8401.160135] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.172060] LustreError: 272406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8401.583527] Lustre: server umount lustre-MDT0001 complete [ 8419.403701] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8422.830978] Key type lgssc unregistered [ 8423.198124] LNet: 276760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8423.207017] LNetError: 276760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8423.222472] LNet: Removed LNI 192.168.204.119@tcp [ 8424.410963] Key type .llcrypt unregistered [ 8424.425168] Key type ._llcrypt unregistered [ 8448.441574] Key type ._llcrypt registered [ 8448.443231] Key type .llcrypt registered [ 8448.614070] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8450.274501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8450.307412] alg: No test for adler32 (adler32-zlib) [ 8451.357276] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 8451.709362] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 8453.432468] Key type lgssc registered [ 8454.857449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8465.056489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8465.084699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.661918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.519345] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8477.154901] LustreError: 277993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.174141] LustreError: 277993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8482.277791] LustreError: 277995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8483.426671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.988447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8489.824574] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8498.810859] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8505.789749] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8514.172171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8514.594790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.614606] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8530.822367] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.211478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8558.054646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.075479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8558.083229] Lustre: Skipped 1 previous similar message [ 8558.088074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8558.100452] Lustre: Skipped 1 previous similar message [ 8562.541844] Lustre: server umount lustre-OST0000 complete [ 8568.292204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8568.303525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8573.103861] Lustre: server umount lustre-MDT0000 complete [ 8573.413240] LustreError: 277996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.435772] LustreError: 277996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8577.507707] LustreError: 277983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.528731] LustreError: 277983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8577.657077] LustreError: 277959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855556 with bad export cookie 6913136990228883041 [ 8577.660626] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.668543] LustreError: 277959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8578.107508] Lustre: server umount lustre-MDT0001 complete [ 8591.671201] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8594.510198] Key type lgssc unregistered [ 8594.805305] LNet: 281715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8594.821770] LNetError: 281715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8595.882185] LNet: Removed LNI 192.168.204.119@tcp [ 8596.853995] Key type .llcrypt unregistered [ 8596.855856] Key type ._llcrypt unregistered [ 8611.679826] Key type ._llcrypt registered [ 8611.681904] Key type .llcrypt registered [ 8611.864425] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8635.314383] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 8653.888927] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 8654.692913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8654.897136] alg: No test for adler32 (adler32-zlib) [ 8655.896677] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 8656.144052] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 8657.816322] Key type lgssc registered [ 8658.977723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8699.243075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8699.290288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.688654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8700.728069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8700.949752] Lustre: lustre-MDT0000: new disk, initializing [ 8701.099282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.164266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8707.520234] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8721.156446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.277831] Lustre: 285265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8721.346805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8721.356891] Lustre: Skipped 1 previous similar message [ 8721.450605] Lustre: lustre-MDT0001: new disk, initializing [ 8721.549613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.607910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8721.624406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8727.938141] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8739.758504] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8748.143750] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8758.430123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.744982] Lustre: lustre-OST0000: new disk, initializing [ 8758.757391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8758.768718] Lustre: 287238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8758.933729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.534113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8764.571239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8764.727512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8767.872382] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8778.919426] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8784.727720] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.963592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8790.077902] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8790.408808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8795.617888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.626857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.637778] LustreError: Skipped 1 previous similar message [ 8795.639361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8795.665924] Lustre: Skipped 1 previous similar message [ 8798.737203] Lustre: server umount lustre-OST0000 complete [ 8803.810903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8803.826054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.841935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.850082] Lustre: Skipped 1 previous similar message [ 8805.860506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.863675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.997692] Lustre: server umount lustre-MDT0000 complete [ 8814.783795] LustreError: 285257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855793 with bad export cookie 13031231389902258598 [ 8814.793453] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8815.327988] Lustre: server umount lustre-MDT0001 complete [ 8826.838273] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:23:22 (1788855802) [ 8837.620591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.167345] LustreError: 289649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.195706] LustreError: 289649:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8838.305404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.538712] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8843.745952] LustreError: 289650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.842666] LustreError: 289649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.968131] LustreError: 289650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8853.497491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.098929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8859.860967] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8870.100125] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8878.438280] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8888.186990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.566894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.067814] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 8908.243814] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8923.135038] Lustre: server umount lustre-OST0000 complete [ 8928.759405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.778883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.780904] Lustre: Skipped 1 previous similar message [ 8928.798886] Lustre: Skipped 2 previous similar messages [ 8933.864502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.871704] Lustre: Skipped 1 previous similar message [ 8938.005822] Lustre: server umount lustre-MDT0000 complete [ 8938.992761] LustreError: 290378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8939.024967] LustreError: 290378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8940.001677] Lustre: 283490:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855902/real 1788855902] req@00000000c4c2988a x1875751085358080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788855918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8940.047371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.097669] Lustre: 283490:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855903/real 1788855903] req@00000000c7200f8d x1875751085358464/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788855919 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8941.984971] LustreError: 289630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855920 with bad export cookie 13031231389902259480 [ 8941.989825] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8941.993542] LustreError: 289630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8942.246800] Lustre: server umount lustre-MDT0001 complete [ 8957.350048] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 8960.539286] Key type lgssc unregistered [ 8960.856117] LNet: 293186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8960.864182] LNetError: 293186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8960.878726] LNet: Removed LNI 192.168.204.119@tcp [ 8961.786074] Key type .llcrypt unregistered [ 8961.787689] Key type ._llcrypt unregistered [ 8991.753728] Key type ._llcrypt registered [ 8991.755437] Key type .llcrypt registered [ 8991.912465] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 9011.066827] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9012.915785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9013.070625] alg: No test for adler32 (adler32-zlib) [ 9014.054616] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 9014.275354] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 9015.952262] Key type lgssc registered [ 9017.348646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9081.664647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9081.719861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.137406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9083.206727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9083.346595] Lustre: lustre-MDT0000: new disk, initializing [ 9083.499644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9083.527297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9089.100756] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9100.556936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.678528] Lustre: 297018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9100.748263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9100.761724] Lustre: Skipped 1 previous similar message [ 9100.890941] Lustre: lustre-MDT0001: new disk, initializing [ 9101.026975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9101.113891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9101.124529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.001625] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9117.879281] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9124.530941] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9131.997761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.410733] Lustre: lustre-OST2710: new disk, initializing [ 9132.414973] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9132.425466] Lustre: 298986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9132.535393] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9140.274818] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9140.855398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9140.873575] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9141.001137] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9151.736172] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.322899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9160.487597] Lustre: lustre-OST03e8: new disk, initializing [ 9160.493259] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9160.500281] Lustre: 300051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9160.649810] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9168.077924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9168.104929] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9168.284284] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9169.102897] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9180.535406] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9189.931546] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9190.239833] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9197.103328] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9197.458994] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9202.593061] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9202.828915] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9208.431164] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9208.725908] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9213.927803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.939327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.954805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9214.443090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.458931] Lustre: Skipped 1 previous similar message [ 9214.467488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9214.479406] Lustre: Skipped 3 previous similar messages [ 9215.707489] Lustre: server umount lustre-MDT0000 complete [ 9219.946404] LustreError: 298985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856198 with bad export cookie 9485627077171747448 [ 9219.955319] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9219.968947] LustreError: 298985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9220.341781] Lustre: server umount lustre-MDT0001 complete [ 9235.037596] Lustre: server umount lustre-OST2710 complete [ 9249.329636] Lustre: server umount lustre-OST03e8 complete [ 9271.435720] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 9284.069904] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9331.174854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.407516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9331.434718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9331.492549] Lustre: lustre-MDT0000: new disk, initializing [ 9331.584334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.607813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9336.801815] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9348.919943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.136366] Lustre: 305304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9349.246546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9349.257242] Lustre: Skipped 1 previous similar message [ 9349.407352] Lustre: lustre-MDT0001: new disk, initializing [ 9349.524468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9349.571443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9349.585763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9355.501935] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9367.952973] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9376.666611] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9385.875031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9386.191869] Lustre: lustre-OST0000: new disk, initializing [ 9386.199665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9386.212357] Lustre: 307274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9386.341809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9393.527290] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9396.302821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9396.317558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9396.387572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9405.006853] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9411.122058] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9411.411975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9416.780767] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9417.211174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9422.320081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9422.322258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.337505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9422.383403] Lustre: Skipped 2 previous similar messages [ 9426.400132] Lustre: server umount lustre-OST0000 complete [ 9431.538082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9431.558655] LustreError: Skipped 1 previous similar message [ 9431.574115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9431.625339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9431.639454] Lustre: Skipped 1 previous similar message [ 9437.670779] LustreError: 305316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9437.678784] LustreError: 305316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9437.891780] Lustre: server umount lustre-MDT0000 complete [ 9442.794586] LustreError: 305315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9442.822296] LustreError: 305315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9442.837903] LustreError: 306098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856421 with bad export cookie 9485627077171749128 [ 9442.852445] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9443.389803] Lustre: server umount lustre-MDT0001 complete [ 9454.078919] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:33:50 (1788856430) [ 9455.902724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9458.156278] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:33:54 (1788856434) [ 9463.100996] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9503.190238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.523766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9503.541384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9503.803849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.160835] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9518.648829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.750053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9518.793981] Lustre: 311134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9524.677302] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9547.966341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9548.332355] Lustre: Denying initial registration attempt from nid 192.168.204.119@tcp, specified as failover [ 9548.353107] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9548.362980] LustreError: 312586:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9548.368536] LustreError: 312586:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9548.375865] LustreError: 312586:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9548.493836] Lustre: server umount lustre-OST0000 complete [ 9548.496039] LustreError: 312586:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9557.473514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.485190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.506274] Lustre: Skipped 2 previous similar messages [ 9557.531173] Lustre: Skipped 3 previous similar messages [ 9562.899539] Lustre: server umount lustre-MDT0000 complete [ 9567.188866] LustreError: 311127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856545 with bad export cookie 9485627077171750010 [ 9567.204569] LustreError: 311127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9567.217958] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9567.549283] Lustre: server umount lustre-MDT0001 complete [ 9579.104231] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 9581.766858] Key type lgssc unregistered [ 9582.092830] LNet: 313706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9582.104227] LNetError: 313706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9582.117388] LNet: Removed LNI 192.168.204.119@tcp [ 9582.971458] Key type .llcrypt unregistered [ 9582.975706] Key type ._llcrypt unregistered [ 9595.927837] Key type ._llcrypt registered [ 9595.930942] Key type .llcrypt registered [ 9596.035458] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9597.038734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9597.140465] alg: No test for adler32 (adler32-zlib) [ 9598.247434] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 9598.608082] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 9600.313062] Key type lgssc registered [ 9601.624865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9658.877498] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9670.526765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9670.540926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.957534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9672.009435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9672.313810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.555669] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9688.656433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.720627] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9688.746920] Lustre: 316471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9689.075445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9694.047921] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9702.852574] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9710.221234] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9731.385935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9731.781118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9732.048674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.134676] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9748.275821] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9753.057472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.066804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.090328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9753.580092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.585327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.608382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9756.572110] Lustre: server umount lustre-OST0000 complete [ 9763.817650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.828834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9763.842246] Lustre: Skipped 1 previous similar message [ 9763.860072] Lustre: Skipped 1 previous similar message [ 9767.198143] Lustre: server umount lustre-MDT0000 complete [ 9772.797898] LustreError: 316461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856751 with bad export cookie 16424376673000750134 [ 9772.809901] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.824407] LustreError: 316461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9773.402594] Lustre: server umount lustre-MDT0001 complete [ 9786.654597] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 9790.601259] Key type lgssc unregistered [ 9790.970426] LNet: 320046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9790.978654] LNetError: 320046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9791.008674] LNet: Removed LNI 192.168.204.119@tcp [ 9792.273973] Key type .llcrypt unregistered [ 9792.281279] Key type ._llcrypt unregistered [ 9825.379891] Key type ._llcrypt registered [ 9825.385499] Key type .llcrypt registered [ 9825.524168] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 9827.165993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9827.340416] alg: No test for adler32 (adler32-zlib) [ 9828.480478] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [ 9828.715440] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 9830.368311] Key type lgssc registered [ 9832.152052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9841.628687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9841.694834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.274631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9848.060149] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9852.386030] LustreError: 321461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.411190] LustreError: 321461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9857.522102] LustreError: 321462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9858.883513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.321295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9864.814572] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9874.110689] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9881.302113] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9892.598505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9893.093209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9900.510701] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9909.102316] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9925.972781] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 9926.120802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9926.134759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.161546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.164972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.166673] Lustre: Skipped 1 previous similar message [ 9928.186852] Lustre: Skipped 1 previous similar message [ 9930.602291] Lustre: server umount lustre-MDT0000 complete [ 9931.053175] LustreError: 321458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.282536] LustreError: 322434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.294360] LustreError: 322434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9934.974400] LustreError: 321441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856913 with bad export cookie 7794011347259892136 [ 9934.999122] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9935.001179] LustreError: 321441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9936.196854] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 9936.204152] Lustre: Skipped 2 previous similar messages [ 9938.405623] LustreError: 321458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9938.420315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.428692] LustreError: 321458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9938.452495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9941.709093] Lustre: server umount lustre-MDT0001 complete [ 9950.063981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9960.474043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.384945] LustreError: 320770:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000f092cb1 x1875752315299584/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9980.686544] LustreError: 323294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9980.708503] LustreError: 323294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9980.789263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.861428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9985.829696] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 9995.260664] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9995.432868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.909537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10000.680780] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10001.398694] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10009.072182] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10010.912981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10017.626262] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10019.663793] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10031.591909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10031.607360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.640477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10031.651662] Lustre: Skipped 1 previous similar message [10033.785881] Lustre: server umount lustre-OST0000 complete [10041.831742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10041.842360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.860944] Lustre: Skipped 1 previous similar message [10041.867630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10041.872492] Lustre: Skipped 2 previous similar messages [10044.890401] Lustre: server umount lustre-MDT0000 complete [10047.458054] LustreError: 325079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.485813] LustreError: 325079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [10048.992811] LustreError: 324269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857027 with bad export cookie 7794011347259899647 [10048.992812] LustreError: 321441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857027 with bad export cookie 7794011347259899647 [10048.992824] LustreError: 321441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10049.008630] LustreError: 324269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10049.028652] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10049.450815] Lustre: server umount lustre-MDT0001 complete [10060.046145] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [10063.348467] Key type lgssc unregistered [10063.703864] LNet: 327757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10063.724015] LNetError: 327757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10063.747917] LNet: Removed LNI 192.168.204.119@tcp [10064.899561] Key type .llcrypt unregistered [10064.906621] Key type ._llcrypt unregistered [10098.735911] Key type ._llcrypt registered [10098.747772] Key type .llcrypt registered [10099.015124] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10100.369594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10100.406446] alg: No test for adler32 (adler32-zlib) [10101.706093] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [10101.910775] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [10103.584229] Key type lgssc registered [10104.826801] Lustre: Echo OBD driver; http://www.lustre.org/ [10113.241934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10113.265879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.947458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.364597] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10125.304349] LustreError: 329171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10125.342795] LustreError: 329171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10130.404989] LustreError: 329172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.470143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.757266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10136.617745] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10146.832967] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10155.537512] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10161.637586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10161.645778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10161.675798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10164.705633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.708053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10164.752025] Lustre: Skipped 1 previous similar message [10166.032520] Lustre: server umount lustre-MDT0000 complete [10169.826034] LustreError: 330274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.845749] LustreError: 330274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10170.439440] LustreError: 329153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857149 with bad export cookie 13111439760831773348 [10170.452743] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10170.459982] LustreError: 329153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10171.062296] Lustre: server umount lustre-MDT0001 complete [10187.426494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.757770] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10187.836861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10188.148601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10193.076700] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10202.528718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10202.583977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10202.614142] Lustre: 331760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10202.789160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10207.954682] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10216.572611] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10223.994774] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10235.118321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10235.408936] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10235.424985] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10235.432210] LustreError: 333644:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10235.439826] LustreError: 333644:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10235.447159] LustreError: 333644:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10235.543313] Lustre: server umount lustre-OST0000 complete [10235.551135] LustreError: 333644:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10246.653830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10246.953072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10247.176096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10254.372512] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10255.105575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10263.108123] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10273.557728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10273.730615] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10273.905410] Lustre: lustre-OST0001: new disk, initializing [10273.913891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10273.926947] Lustre: 335186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10274.069897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10281.906314] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10283.630502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10283.646512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10283.739989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10289.261354] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10294.266181] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10294.266348] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.283179] LustreError: Skipped 1 previous similar message [10294.284856] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10294.336732] Lustre: Skipped 1 previous similar message [10298.734997] Lustre: server umount lustre-OST0001 complete [10299.366875] LustreError: 334112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10299.384990] LustreError: 334112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10304.483878] LustreError: 334110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.488899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.500140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10304.500148] Lustre: Skipped 1 previous similar message [10304.523048] LustreError: 334110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10304.580746] Lustre: Skipped 1 previous similar message [10309.612315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10309.622513] Lustre: Skipped 2 previous similar messages [10314.723650] LustreError: 335624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10314.766818] LustreError: 335624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10317.280876] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10317.479813] Lustre: server umount lustre-OST0000 complete [10324.963269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.995824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10325.017245] Lustre: Skipped 3 previous similar messages [10336.225752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10336.753515] Lustre: server umount lustre-MDT0000 complete [10340.326940] LustreError: 333646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.364314] LustreError: 333646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10342.585156] LustreError: 335185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857321 with bad export cookie 13111439760831773901 [10342.586545] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10342.613915] LustreError: 335185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10343.099044] Lustre: server umount lustre-MDT0001 complete [10356.474294] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [10358.995422] Key type lgssc unregistered [10359.247533] LNet: 337148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10359.260437] LNetError: 337148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10360.300828] LNet: Removed LNI 192.168.204.119@tcp [10361.154720] Key type .llcrypt unregistered [10361.156440] Key type ._llcrypt unregistered [10441.448361] Key type ._llcrypt registered [10441.453039] Key type .llcrypt registered [10441.659111] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [10459.724411] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10460.878788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10460.938700] alg: No test for adler32 (adler32-zlib) [10462.183090] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [10462.549721] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [10464.248235] Key type lgssc registered [10465.602824] Lustre: Echo OBD driver; http://www.lustre.org/ [10504.502822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10504.553408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.896908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10505.939453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10506.024289] Lustre: lustre-MDT0000: new disk, initializing [10506.139564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10506.159614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10510.243909] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10521.178764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.301922] Lustre: 342079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10521.380214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10521.383629] Lustre: Skipped 1 previous similar message [10521.479379] Lustre: lustre-MDT0001: new disk, initializing [10521.598415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10521.650293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10521.668858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10526.284726] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10536.394625] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10543.420888] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10551.157546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10551.415744] Lustre: lustre-OST0000: new disk, initializing [10551.427330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10551.433391] Lustre: 344048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10551.528069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10559.688156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10559.708113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10559.849715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10560.029056] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10572.323072] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10578.586500] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10578.826553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10584.173607] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10584.483809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10590.694172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.698244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10590.723384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10590.728983] Lustre: Skipped 1 previous similar message [10593.057037] Lustre: server umount lustre-OST0000 complete [10598.370151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10598.375500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.390091] Lustre: Skipped 1 previous similar message [10598.397944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10600.931528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10600.946487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10603.060254] Lustre: server umount lustre-MDT0000 complete [10607.101921] LustreError: 342072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857585 with bad export cookie 11337352233959863571 [10607.113395] LustreError: 342072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10607.116953] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10607.482191] Lustre: server umount lustre-MDT0001 complete [10617.841838] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:53:14 (1788857594) [10626.788464] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:53:23 (1788857603) [10636.599734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.042722] LustreError: 346808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.062288] LustreError: 346808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10637.156550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10641.789972] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10642.408935] LustreError: 346809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10646.503112] LustreError: 346808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.793956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.347866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10655.626579] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10664.114680] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10670.090668] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10676.521667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.880452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10683.816230] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10691.955251] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10704.480471] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10706.387400] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10708.159870] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10709.956118] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10711.690802] Lustre: Failing over lustre-MDT0000 [10711.994083] Lustre: server umount lustre-MDT0000 complete [10712.043860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.048453] LustreError: 346804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10712.059514] Lustre: Skipped 2 previous similar messages [10712.087679] LustreError: 346804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10717.166314] LustreError: 346809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.191382] LustreError: 346809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10725.675562] LustreError: 346805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.712668] LustreError: 346805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10728.419739] Lustre: 340304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788857690/real 1788857690] req@00000000ef1a659c x1875752980126976/t0(0) o400->MGC192.168.204.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788857706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10728.455663] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10731.600344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.679131] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9d565dc9f6a8bc98 [10738.707725] Lustre: MGC192.168.204.119@tcp: Connection restored to 0@lo (at 0@lo) [10738.980579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10739.077965] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10741.036412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10743.620627] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10744.310571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10744.334638] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10744.411595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10754.446508] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.342973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10759.982464] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10762.438114] Lustre: Failing over lustre-MDT0000 [10762.747649] Lustre: server umount lustre-MDT0000 complete [10764.774595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.778017] LustreError: 347536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.778716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10764.785897] Lustre: Skipped 2 previous similar messages [10764.813178] LustreError: 347536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [10781.152168] Lustre: 340304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788857743/real 1788857743] req@000000002aa6638b x1875752980148096/t0(0) o400->MGC192.168.204.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788857759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10781.184372] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10781.844972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.403753] LustreError: 340300:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e52fa8e7 x1875752980154368/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10791.662343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10791.721023] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10792.243174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10795.618714] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10797.033361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10797.039823] Lustre: Skipped 2 previous similar messages [10797.070154] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10797.113731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10804.378741] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10805.761435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10812.392697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10812.395094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10812.396856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10812.396863] Lustre: Skipped 1 previous similar message [10812.400890] Lustre: Skipped 2 previous similar messages [10817.053259] Lustre: server umount lustre-MDT0000 complete [10817.507430] LustreError: 346809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.527305] LustreError: 346809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [10820.443303] LustreError: 346788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857799 with bad export cookie 11337352233959866315 [10820.450632] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.458808] LustreError: 346788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10820.706404] Lustre: server umount lustre-MDT0001 complete [10834.430803] Lustre: server umount lustre-OST0000 complete [10848.032241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.846825] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10861.296509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.621766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10861.632650] Lustre: Skipped 1 previous similar message [10865.292292] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10873.826238] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10881.442644] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10889.650146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.008821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10896.419729] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [10904.421309] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10914.790080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10914.810662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.823397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10914.830023] Lustre: Skipped 2 previous similar messages [10920.396648] Lustre: server umount lustre-OST0000 complete [10928.104544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10928.107412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10928.108703] Lustre: Skipped 3 previous similar messages [10928.121762] LustreError: Skipped 1 previous similar message [10930.331508] Lustre: server umount lustre-MDT0000 complete [10933.219366] LustreError: 353169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.268777] LustreError: 353169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10936.409863] LustreError: 353151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857915 with bad export cookie 11337352233959867036 [10936.415093] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10936.430706] LustreError: 353151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10936.775215] Lustre: server umount lustre-MDT0001 complete [10951.115662] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [10954.044882] Key type lgssc unregistered [10954.391097] LNet: 356703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10954.397657] LNetError: 356703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10954.433037] LNet: Removed LNI 192.168.204.119@tcp [10955.277897] Key type .llcrypt unregistered [10955.284123] Key type ._llcrypt unregistered [10983.063552] Key type ._llcrypt registered [10983.066668] Key type .llcrypt registered [10983.181217] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [10984.181330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10984.368526] alg: No test for adler32 (adler32-zlib) [10985.450874] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [10985.717785] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [10987.409495] Key type lgssc registered [10988.942215] Lustre: Echo OBD driver; http://www.lustre.org/ [10996.930734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10996.945572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.482192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.960689] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11009.000778] LustreError: 358220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.022580] LustreError: 358220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11011.920178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.515658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11017.468362] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11026.168468] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11032.744595] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11039.794229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.303247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11044.420776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11046.866918] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11055.090185] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11066.851507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11066.855136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11066.876293] Lustre: Skipped 2 previous similar messages [11066.897542] Lustre: Skipped 2 previous similar messages [11071.109658] Lustre: server umount lustre-MDT0000 complete [11071.972274] LustreError: 358216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.996127] LustreError: 358216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11075.087662] LustreError: 358201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858053 with bad export cookie 14539379721398694160 [11075.101967] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.105956] LustreError: 358201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11075.735663] Lustre: server umount lustre-MDT0001 complete [11091.029556] Lustre: server umount lustre-OST0000 complete [11103.741436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11104.049510] LustreError: 361944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.114692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11108.772840] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11109.349322] LustreError: 361945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.444608] LustreError: 361944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.679221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11118.027910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11122.137692] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11129.959886] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11136.645488] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11144.188861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11144.641756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11148.727186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11152.009605] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11160.295172] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11173.350417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.351611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.362801] Lustre: Skipped 2 previous similar messages [11173.371069] Lustre: Skipped 2 previous similar messages [11178.466311] LustreError: 361945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11178.470144] Lustre: server umount lustre-MDT0000 complete [11178.476268] LustreError: 361945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11181.749894] LustreError: 361925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858160 with bad export cookie 14539379721398695273 [11181.760643] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11181.761509] LustreError: 361925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11182.065552] Lustre: server umount lustre-MDT0001 complete [11194.740933] Lustre: server umount lustre-OST0000 complete [11210.114950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.511479] LustreError: 365856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11210.590179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.138724] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11223.854864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11228.623166] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11235.804344] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11242.838804] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11250.939734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11251.488994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11251.497547] Lustre: Skipped 1 previous similar message [11255.599392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11259.104434] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11267.598196] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11274.216031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.219545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11274.220975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11274.229983] Lustre: Skipped 1 previous similar message [11275.191533] Lustre: server umount lustre-OST0000 complete [11279.333025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.348979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11279.360730] Lustre: Skipped 2 previous similar messages [11284.453585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.469914] Lustre: Skipped 1 previous similar message [11285.330499] Lustre: server umount lustre-MDT0000 complete [11288.975956] LustreError: 366591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858267 with bad export cookie 14539379721398696302 [11288.982162] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11288.991840] LustreError: 366591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11289.346806] Lustre: server umount lustre-MDT0001 complete [11301.381239] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11304.474618] Key type lgssc unregistered [11304.774726] LNet: 369387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11304.785515] LNetError: 369387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11304.812250] LNet: Removed LNI 192.168.204.119@tcp [11305.634131] Key type .llcrypt unregistered [11305.636260] Key type ._llcrypt unregistered [11333.578247] Key type ._llcrypt registered [11333.579925] Key type .llcrypt registered [11333.771696] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11334.848805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11334.966187] alg: No test for adler32 (adler32-zlib) [11335.980724] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [11336.359058] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11338.160297] Key type lgssc registered [11339.288940] Lustre: Echo OBD driver; http://www.lustre.org/ [11346.955782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11346.982656] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.997651] LustreError: 370824:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11347.025972] LustreError: 370824:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11347.045194] LustreError: 370824:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11347.054314] LustreError: 370824:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11347.066949] LustreError: 370824:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11354.883675] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11354.891279] LustreError: 371222:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11354.913468] LustreError: 371222:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11354.928169] LustreError: 371222:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11354.936287] LustreError: 371222:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11354.946976] LustreError: 371222:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11375.733320] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11378.534850] Key type lgssc unregistered [11378.819591] LNet: 372224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11378.832174] LNetError: 372224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11378.844784] LNet: Removed LNI 192.168.204.119@tcp [11379.741360] Key type .llcrypt unregistered [11379.745827] Key type ._llcrypt unregistered [11397.483238] Key type ._llcrypt registered [11397.486805] Key type .llcrypt registered [11397.606829] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [11413.290900] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11414.364412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11414.403300] alg: No test for adler32 (adler32-zlib) [11415.486667] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [11415.736693] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11417.392295] Key type lgssc registered [11418.536098] Lustre: Echo OBD driver; http://www.lustre.org/ [11451.892097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11451.936121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.213606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11453.239516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11453.353985] Lustre: lustre-MDT0000: new disk, initializing [11453.495572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11453.517225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11458.048273] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11467.715882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.797864] Lustre: 375376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11467.842717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11467.846991] Lustre: Skipped 1 previous similar message [11467.911243] Lustre: lustre-MDT0001: new disk, initializing [11467.991968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11468.021378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11468.032119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11471.416369] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11479.991987] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11485.603062] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11492.102466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.346040] Lustre: lustre-OST0000: new disk, initializing [11492.349748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11492.355428] Lustre: 377345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11492.436989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11498.053334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11498.061705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11498.177600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11498.670989] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11507.942844] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11512.104581] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11512.307360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11516.634866] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11516.853607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11518.960521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11518.963243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11518.968444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11518.972178] LustreError: Skipped 1 previous similar message [11518.992892] Lustre: Skipped 1 previous similar message [11524.075540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11524.083938] Lustre: Skipped 2 previous similar messages [11529.206489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11529.215327] Lustre: Skipped 1 previous similar message [11533.280692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11533.436298] Lustre: server umount lustre-OST0000 complete [11539.429189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.437102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.468526] Lustre: Skipped 1 previous similar message [11539.479906] Lustre: Skipped 1 previous similar message [11543.859944] Lustre: server umount lustre-MDT0000 complete [11547.566091] LustreError: 375369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858526 with bad export cookie 5300190402510059348 [11547.567762] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11547.575893] LustreError: 375369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11547.877666] Lustre: server umount lustre-MDT0001 complete [11556.502751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11557.926473] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:08:55 (1788858535) [11564.198684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.803089] LustreError: 379807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11564.831473] LustreError: 379807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11564.902492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11569.196299] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11570.152827] LustreError: 379808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.267000] LustreError: 379807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.575715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.998468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11582.188963] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11589.770430] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11595.082688] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11601.456549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11601.735866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11607.877124] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11615.431245] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11622.388371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11622.579643] Lustre: lustre-OST0001: new disk, initializing [11622.583921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11622.589311] Lustre: 382643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11622.669479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11629.270764] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11631.210877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11631.216903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11631.358094] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11640.143693] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11646.946788] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.949445] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11646.960524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11646.981677] Lustre: Skipped 1 previous similar message [11650.234510] Lustre: server umount lustre-OST0001 complete [11652.071533] LustreError: 383078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.085441] LustreError: 383078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11657.194408] LustreError: 383078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.211526] LustreError: 383078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11667.435905] LustreError: 381630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.450773] LustreError: 381630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11669.476895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11669.488453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.512457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11669.523441] Lustre: Skipped 2 previous similar messages [11671.521423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11682.272178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11682.546587] Lustre: server umount lustre-OST0000 complete [11687.910753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.912731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.939665] Lustre: Skipped 2 previous similar messages [11687.964495] Lustre: Skipped 6 previous similar messages [11692.969390] Lustre: server umount lustre-MDT0000 complete [11693.031911] LustreError: 379802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.050044] LustreError: 379802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11696.821517] LustreError: 379787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858675 with bad export cookie 5300190402510060230 [11696.823227] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11696.841294] LustreError: 379787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11703.220229] Lustre: server umount lustre-MDT0001 complete [11716.238073] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [11719.210136] Key type lgssc unregistered [11719.527432] LNet: 384631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11719.532917] LNetError: 384631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11719.550765] LNet: Removed LNI 192.168.204.119@tcp [11720.426849] Key type .llcrypt unregistered [11720.439556] Key type ._llcrypt unregistered [11770.797691] Key type ._llcrypt registered [11770.799605] Key type .llcrypt registered [11770.952657] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [11772.122392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11772.147065] alg: No test for adler32 (adler32-zlib) [11773.176754] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [11773.410056] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [11775.064289] Key type lgssc registered [11775.961484] Lustre: Echo OBD driver; http://www.lustre.org/ [11783.959319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11783.995080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11785.201939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11785.223296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11785.347274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11789.253582] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11798.751031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.833870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11798.875089] Lustre: 387432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11799.164382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11802.980610] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11809.466492] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11816.769599] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11819.492328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11819.500810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.517073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.031862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.032554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11826.146237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11826.150353] Lustre: Skipped 2 previous similar messages [11831.271685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11831.279868] Lustre: Skipped 1 previous similar message [11833.312334] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11833.571969] Lustre: server umount lustre-MDT0000 complete [11837.384301] LustreError: 387423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858816 with bad export cookie 7251806227809402213 [11837.386744] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11837.395621] LustreError: 387423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11837.680724] Lustre: server umount lustre-MDT0001 complete [11846.614616] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:13:43 (1788858823) [11852.498056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.133078] LustreError: 389944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.152810] LustreError: 389944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11853.209675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.336382] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11858.404646] LustreError: 389945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.521127] LustreError: 389944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.313907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.656248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11868.875121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11873.967151] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11882.942673] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11888.971347] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11896.622586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11896.904556] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11897.096961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11903.773860] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11912.243736] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11919.526643] Lustre: Permanently deactivating lustre-OST0000 [11919.541414] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11926.083948] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11928.843857] LustreError: 392934:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11928.851483] LustreError: 392934:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11928.857024] LustreError: 392934:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11931.596410] LustreError: 392982:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11931.605094] LustreError: 392982:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11939.365325] Lustre: server umount lustre-OST0000 complete [11941.991582] LustreError: 393232:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11941.999488] LustreError: 393232:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11945.956596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.957431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11945.973157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.985744] Lustre: Skipped 1 previous similar message [11958.240733] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11958.443786] Lustre: server umount lustre-MDT0000 complete [11961.315282] LustreError: 390938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.333997] LustreError: 390938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11961.898525] LustreError: 391781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858940 with bad export cookie 7251806227809402794 [11961.902887] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.915369] LustreError: 391781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11971.714034] Lustre: server umount lustre-MDT0001 complete [11979.723855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11983.559218] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [11985.995160] LustreError: 394392:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11991.708256] Lustre: 394731:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.119@tcp [11993.598376] Lustre: server umount MGS complete [12004.740903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12005.236210] LustreError: 395456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12005.286206] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12005.295172] Lustre: Skipped 1 previous similar message [12005.405789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12009.846190] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12015.588989] LustreError: 395457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.602831] LustreError: 395457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12019.035471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.462529] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12023.998751] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12032.940917] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12040.239212] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12042.014772] Lustre: Permanently reactivating lustre-OST0000 [12042.025612] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12042.031888] Lustre: Skipped 2 previous similar messages [12072.416375] LustreError: 397285:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12081.559817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12082.076189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12082.089213] Lustre: Skipped 1 previous similar message [12089.573534] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12099.429804] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12107.438842] Lustre: server umount lustre-OST0000 complete [12111.843128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12111.862572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.887461] Lustre: Skipped 1 previous similar message [12111.892465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12111.902892] Lustre: Skipped 4 previous similar messages [12121.194979] Lustre: server umount lustre-MDT0000 complete [12124.138182] LustreError: 397138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.168215] LustreError: 397138:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12124.962210] LustreError: 395438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859103 with bad export cookie 7251806227809403844 [12124.968430] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12124.971928] LustreError: 395438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12135.579334] Lustre: server umount lustre-MDT0001 complete [12147.821489] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12150.657087] Key type lgssc unregistered [12150.914670] LNet: 399870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12150.920790] LNetError: 399870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.941703] LNet: Removed LNI 192.168.204.119@tcp [12151.883748] Key type .llcrypt unregistered [12151.885628] Key type ._llcrypt unregistered [12170.943555] Key type ._llcrypt registered [12170.945674] Key type .llcrypt registered [12171.077930] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [12187.218959] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12188.146468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12188.163667] alg: No test for adler32 (adler32-zlib) [12189.210297] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [12189.494634] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12191.176252] Key type lgssc registered [12192.478624] Lustre: Echo OBD driver; http://www.lustre.org/ [12227.764979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12227.800644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12229.094092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12229.118501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12229.202544] Lustre: lustre-MDT0000: new disk, initializing [12229.324775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12229.352237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12233.930395] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12244.731154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12244.965948] Lustre: 403023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12245.090444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12245.097339] Lustre: Skipped 1 previous similar message [12245.232543] Lustre: lustre-MDT0001: new disk, initializing [12245.362231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12245.423820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12245.444882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12250.035555] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12260.053094] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12266.713966] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12272.962057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12273.213616] Lustre: lustre-OST0000: new disk, initializing [12273.217979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12273.224938] Lustre: 404991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12273.354838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12278.760899] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12280.397310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12280.409784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12280.532628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12288.331539] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12292.384661] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12292.583517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12297.093864] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12297.343988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12301.285781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12301.289635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.293412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.315564] Lustre: Skipped 1 previous similar message [12301.334728] Lustre: Skipped 1 previous similar message [12305.468718] Lustre: server umount lustre-OST0000 complete [12311.523561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12311.534475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12311.546971] Lustre: Skipped 1 previous similar message [12311.562287] Lustre: Skipped 1 previous similar message [12315.621621] Lustre: server umount lustre-MDT0000 complete [12319.217580] LustreError: 403828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859297 with bad export cookie 2852677363288167125 [12319.223578] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12319.232635] LustreError: 403828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12319.591275] Lustre: server umount lustre-MDT0001 complete [12328.170989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12330.071634] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:21:46 (1788859306) [12337.793685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.297951] LustreError: 407452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.317281] LustreError: 407452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12338.374966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12343.215758] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12343.784975] LustreError: 407453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.888656] LustreError: 407452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12352.172205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12352.537907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12357.367813] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12365.791916] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12372.066170] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12378.095519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12378.369690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12383.906092] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12390.734425] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12392.541917] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12395.001545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12408.302125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.308034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12408.314840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12408.333057] Lustre: Skipped 1 previous similar message [12408.381492] LustreError: Skipped 1 previous similar message [12411.778737] Lustre: server umount lustre-OST0000 complete [12418.530642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.535867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12418.552886] Lustre: Skipped 1 previous similar message [12418.565298] Lustre: Skipped 2 previous similar messages [12422.528101] Lustre: server umount lustre-MDT0000 complete [12423.649362] LustreError: 407453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12423.691550] LustreError: 407453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12427.813785] LustreError: 407432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859406 with bad export cookie 2852677363288168007 [12427.849036] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12428.888493] Lustre: server umount lustre-MDT0001 complete [12445.306154] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12448.776898] Key type lgssc unregistered [12449.126796] LNet: 411176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12449.145194] LNetError: 411176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12449.163646] LNet: Removed LNI 192.168.204.119@tcp [12450.337112] Key type .llcrypt unregistered [12450.341081] Key type ._llcrypt unregistered [12473.970152] Key type ._llcrypt registered [12473.972416] Key type .llcrypt registered [12474.057292] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12474.352793] Key type .llcrypt unregistered [12474.355660] Key type ._llcrypt unregistered [12491.619327] Key type ._llcrypt registered [12491.622471] Key type .llcrypt registered [12491.708879] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12493.077618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12493.125452] alg: No test for adler32 (adler32-zlib) [12494.440243] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [12494.805862] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12496.528336] Key type lgssc registered [12498.025443] Lustre: Echo OBD driver; http://www.lustre.org/ [12506.026235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12506.040826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.539593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12512.485841] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12517.860315] LustreError: 413381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.889813] LustreError: 413381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12522.400548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12522.918825] LustreError: 413382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12522.960836] LustreError: 413382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12522.975147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12528.098568] LustreError: 413381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.122042] LustreError: 413381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12530.334428] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12533.222569] LustreError: 413378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.242907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12533.250792] LustreError: 413378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12538.340694] LustreError: 413378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.354636] LustreError: 413378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12538.611819] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12545.504149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.881384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12550.176746] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12561.386741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.398943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.407941] Lustre: Skipped 1 previous similar message [12561.430195] Lustre: Skipped 1 previous similar message [12566.504985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12571.626981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12571.645996] Lustre: Skipped 2 previous similar messages [12574.177035] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12574.509736] Lustre: server umount lustre-OST0000 complete [12581.863124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12581.863634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12581.865350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12581.889362] Lustre: Skipped 1 previous similar message [12584.840911] Lustre: server umount lustre-MDT0000 complete [12586.977446] LustreError: 413377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12586.995252] LustreError: 413377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12588.355504] LustreError: 413361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859566 with bad export cookie 4914922730965297017 [12588.357057] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12588.364738] LustreError: 413361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12588.735623] Lustre: server umount lustre-MDT0001 complete [12599.198562] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12601.613866] Key type lgssc unregistered [12601.874105] LNet: 416530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12601.880483] LNetError: 416530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12601.896589] LNet: Removed LNI 192.168.204.119@tcp [12602.581706] Key type .llcrypt unregistered [12602.584113] Key type ._llcrypt unregistered [12624.321713] Key type ._llcrypt registered [12624.325579] Key type .llcrypt registered [12624.456917] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12625.334267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12625.415722] alg: No test for adler32 (adler32-zlib) [12626.544455] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [12626.800219] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12628.488617] Key type lgssc registered [12629.849937] Lustre: Echo OBD driver; http://www.lustre.org/ [12638.720465] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12638.736508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12654.304545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12659.424432] LustreError: 417919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [12686.049067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12693.124160] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12700.785198] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12708.492409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.967867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12709.035087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12713.590530] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12713.957558] LustreError: 417945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.989545] LustreError: 417945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12719.076218] LustreError: 418977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.143108] LustreError: 418954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.179069] LustreError: 418954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12724.530549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12725.118404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12730.829446] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12740.056557] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12746.992724] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12754.914970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12754.931118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.969244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12755.943817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12755.950368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12755.986959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12761.059262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12761.066370] Lustre: Skipped 1 previous similar message [12766.179225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12766.194769] Lustre: Skipped 1 previous similar message [12768.224207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12768.390544] Lustre: server umount lustre-OST0000 complete [12776.420016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12776.426849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12776.432076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12776.449412] Lustre: Skipped 1 previous similar message [12778.880313] Lustre: server umount lustre-MDT0000 complete [12781.537262] LustreError: 419535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12781.553269] LustreError: 419535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12782.618813] LustreError: 417927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859761 with bad export cookie 9806652371647352560 [12782.636789] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12783.024905] Lustre: server umount lustre-MDT0001 complete [12793.996465] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12797.550769] Key type lgssc unregistered [12797.929981] LNet: 421492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12797.944902] LNetError: 421492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12797.970661] LNet: Removed LNI 192.168.204.119@tcp [12799.072152] Key type .llcrypt unregistered [12799.074751] Key type ._llcrypt unregistered [12827.297541] Key type ._llcrypt registered [12827.300830] Key type .llcrypt registered [12827.411061] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12828.246374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12828.271152] alg: No test for adler32 (adler32-zlib) [12829.241250] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [12829.481668] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12831.152569] Key type lgssc registered [12832.278192] Lustre: Echo OBD driver; http://www.lustre.org/ [12840.243701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12840.256742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.698201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12846.584401] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12852.207504] LustreError: 422903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.229697] LustreError: 422903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12855.483919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12856.091682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12862.659252] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12871.276737] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12877.157275] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12884.494895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12884.909246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12886.981505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12890.830972] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12898.109617] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12906.327325] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.19@tcp (stopping) [12906.468927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.494580] Lustre: Skipped 2 previous similar messages [12909.403031] Lustre: server umount lustre-MDT0000 complete [12911.403652] LustreError: 422900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12912.556616] LustreError: 422885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.19@tcp arrived at 1788859891 with bad export cookie 11967408633216270813 [12914.745510] LustreError: 422886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859893 with bad export cookie 11967408633216269924 [12914.758446] LustreError: 422886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12914.758681] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12916.706516] LustreError: 423631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.710343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12916.710529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.726021] LustreError: 423631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12916.729317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12916.766603] Lustre: Skipped 3 previous similar messages [12919.714170] LustreError: 422903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12919.720472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.746026] LustreError: 422903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12920.949076] Lustre: server umount lustre-OST0000 complete [12924.836675] LustreError: 423631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.862253] LustreError: 423631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12927.801179] Lustre: server umount lustre-MDT0001 complete [12939.496600] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [12942.524535] Key type lgssc unregistered [12942.892856] LNet: 426590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12942.904490] LNetError: 426590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12942.931917] LNet: Removed LNI 192.168.204.119@tcp [12944.077753] Key type .llcrypt unregistered [12944.081365] Key type ._llcrypt unregistered [12971.504935] Key type ._llcrypt registered [12971.507247] Key type .llcrypt registered [12971.603095] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [12972.910921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12972.986728] alg: No test for adler32 (adler32-zlib) [12974.063657] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [12974.340572] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [12976.056305] Key type lgssc registered [12977.328917] Lustre: Echo OBD driver; http://www.lustre.org/ [12985.242948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12985.273427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.807536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12992.922092] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [12997.090445] LustreError: 428022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12997.109530] LustreError: 428022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13002.217753] LustreError: 428023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.342526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.239831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13012.239118] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13020.864695] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.469313] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13038.264728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13038.719466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13045.519370] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13054.186105] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13058.246620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13063.660708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.680880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13063.687662] Lustre: Skipped 1 previous similar message [13065.558262] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [13068.774800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13068.791165] Lustre: Skipped 1 previous similar message [13069.598718] Lustre: server umount lustre-MDT0001 complete [13070.635614] LustreError: 428019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13073.894444] LustreError: 428019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13073.920426] LustreError: 428019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13079.011265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13079.012185] LustreError: 428023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13079.021202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13079.065928] Lustre: Skipped 1 previous similar message [13079.068889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13083.216887] Lustre: server umount lustre-OST0000 complete [13087.687479] Lustre: server umount lustre-MDT0000 complete [13099.863813] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13102.619369] Key type lgssc unregistered [13102.940150] LNet: 431709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13102.956489] LNetError: 431709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13102.976263] LNet: Removed LNI 192.168.204.119@tcp [13103.835472] Key type .llcrypt unregistered [13103.838046] Key type ._llcrypt unregistered [13129.736782] Key type ._llcrypt registered [13129.739698] Key type .llcrypt registered [13129.826205] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [13145.318665] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13146.284568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13146.392342] alg: No test for adler32 (adler32-zlib) [13147.475231] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [13147.786402] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [13149.488258] Key type lgssc registered [13150.658343] Lustre: Echo OBD driver; http://www.lustre.org/ [13191.826257] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13207.925066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13207.969055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13209.268850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13209.326963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13209.397828] Lustre: lustre-MDT0000: new disk, initializing [13209.474564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13209.490726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13213.765077] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13229.555057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13229.808871] Lustre: lustre-OST0000: new disk, initializing [13229.817987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13229.824311] Lustre: Skipped 1 previous similar message [13229.830646] Lustre: 436741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13229.913104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13236.194870] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13236.825091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13236.836180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13236.869158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13245.021876] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13261.136041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13261.261518] Lustre: 435723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13261.337597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13261.457462] Lustre: lustre-MDT0001: new disk, initializing [13261.588477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13261.650384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13261.659090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13267.012837] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13276.790703] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13277.046399] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13281.598069] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13281.768798] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13287.749928] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [13287.910983] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13287.916690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13287.941047] Lustre: Skipped 1 previous similar message [13292.510420] Lustre: server umount lustre-MDT0001 complete [13298.149744] LustreError: 438823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.165439] LustreError: 438823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13303.270612] LustreError: 435734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13303.285986] LustreError: 435734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13308.385784] LustreError: 435735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13308.423118] LustreError: 435735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13313.506417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13313.506657] LustreError: 435734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13313.514725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13313.517470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13313.517479] Lustre: Skipped 2 previous similar messages [13318.626950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13318.632454] LustreError: 435735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13323.752444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13325.280255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13325.585956] Lustre: server umount lustre-OST0000 complete [13328.870500] LustreError: 435734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13328.888388] LustreError: 435734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13336.156683] Lustre: server umount lustre-MDT0000 complete [13352.412471] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13355.298514] Key type lgssc unregistered [13355.546605] LNet: 440492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13355.554413] LNetError: 440492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13355.568185] LNet: Removed LNI 192.168.204.119@tcp [13356.575319] Key type .llcrypt unregistered [13356.579095] Key type ._llcrypt unregistered [13399.628618] Key type ._llcrypt registered [13399.631140] Key type .llcrypt registered [13399.721599] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [13400.074550] Key type .llcrypt unregistered [13400.078737] Key type ._llcrypt unregistered [13419.405322] Key type ._llcrypt registered [13419.408375] Key type .llcrypt registered [13419.495337] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [13421.234650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13421.280982] alg: No test for adler32 (adler32-zlib) [13422.494929] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [13422.855937] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [13424.536593] Key type lgssc registered [13426.044810] Lustre: Echo OBD driver; http://www.lustre.org/ [13459.304845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13459.324059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13460.523936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13460.581432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13460.686494] Lustre: lustre-MDT0000: new disk, initializing [13460.778216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13460.825229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13466.674520] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13479.980645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13480.218782] Lustre: 443865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13480.363376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13480.386361] Lustre: Skipped 1 previous similar message [13480.518746] Lustre: lustre-MDT0001: new disk, initializing [13480.676436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13480.749263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13480.769757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13486.269898] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13497.528742] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13504.871832] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13513.276640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13513.753163] Lustre: lustre-OST0000: new disk, initializing [13513.762562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13513.789329] Lustre: 445833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13514.017094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13520.510848] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13522.051176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13522.067461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13522.182953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13530.697191] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13542.371804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13542.382817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13542.403569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13542.886367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13542.901872] Lustre: Skipped 1 previous similar message [13542.913653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13542.917334] Lustre: Skipped 1 previous similar message [13545.904905] Lustre: server umount lustre-MDT0000 complete [13550.066507] LustreError: 443855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860528 with bad export cookie 17014957205003264298 [13550.082395] LustreError: 443855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13550.084434] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13550.562751] Lustre: server umount lustre-MDT0001 complete [13565.222308] Lustre: server umount lustre-OST0000 complete [13577.624612] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:42:34 (1788860554) [13586.784394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13587.513874] LustreError: 447864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13587.539292] LustreError: 447864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13587.594595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13592.163960] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13593.057870] LustreError: 447865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13598.181355] LustreError: 447864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13601.306443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13602.035075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13607.017941] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13616.005767] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13621.598594] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13641.856202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13642.096467] Lustre: Found index 0 for lustre-OST0000, updating log [13642.278659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13648.994566] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [13658.009205] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13658.686125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13668.840723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13668.845608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13668.846346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13668.864358] Lustre: Skipped 1 previous similar message [13668.901333] Lustre: Skipped 1 previous similar message [13674.427168] Lustre: server umount lustre-OST0000 complete [13691.364282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13691.392458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13691.399212] Lustre: Skipped 3 previous similar messages [13695.524248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13695.536656] Lustre: Skipped 1 previous similar message [13696.552434] LustreError: 447864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13696.662635] Lustre: server umount lustre-MDT0000 complete [13700.111030] LustreError: 449720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860678 with bad export cookie 17014957205003265691 [13700.111975] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13700.127828] LustreError: 449720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13700.430199] Lustre: server umount lustre-MDT0001 complete [13708.652692] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:44:45 (1788860685) [13710.752300] LNet: 1 local NIs in recovery (showing 1): 192.168.204.119@tcp [14331.383098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14331.769471] LustreError: 454280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14331.790721] LustreError: 454280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14331.936125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14336.995989] LustreError: 454281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14337.620153] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14342.116132] LustreError: 454280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14345.868554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14346.300445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14352.530091] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14361.085967] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14367.556332] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14387.501311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14387.834392] Lustre: Found index 0 for lustre-OST0000, updating log [14388.157349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14389.664179] LNet: 1 local NIs in recovery (showing 1): 192.168.204.119@tcp [14395.503397] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14404.399807] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14407.543786] Lustre: server umount lustre-OST0000 complete [14412.769594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14412.782869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14412.810049] Lustre: Skipped 1 previous similar message [14412.812939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14413.860141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14413.901075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14415.971899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14415.980804] Lustre: Skipped 4 previous similar messages [14418.374300] Lustre: server umount lustre-MDT0000 complete [14419.046511] LustreError: 454276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14419.070779] LustreError: 454276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14423.218057] LustreError: 454275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14423.271503] LustreError: 454275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [14424.342697] LustreError: 454260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861402 with bad export cookie 17014957205003266776 [14424.352405] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14424.372403] LustreError: 454260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14424.876711] Lustre: server umount lustre-MDT0001 complete [14450.145209] LNet: 1 local NIs in recovery (showing 1): 192.168.204.119@tcp [14458.230595] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [14468.184182] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [14503.013563] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:58:00 (1788861480) [14519.701996] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [14522.680790] Key type lgssc unregistered [14523.018922] LNet: 460900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14523.030624] LNetError: 460900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14649.000991] LNet: Removed LNI 192.168.204.119@tcp [14649.946536] Key type .llcrypt unregistered [14649.948719] Key type ._llcrypt unregistered [14664.650849] Key type ._llcrypt registered [14664.652929] Key type .llcrypt registered [14664.733024] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [14665.568370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14665.608311] alg: No test for adler32 (adler32-zlib) [14666.627651] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [14666.842383] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [14668.520308] Key type lgssc registered [14669.490388] Lustre: Echo OBD driver; http://www.lustre.org/ [14673.140680] LNet: Added LNI 192.168.204.119@tcp100 [8/256/0/180] [14673.860470] LNet: Added LNI 192.168.204.119@tcp101 [8/256/0/180] [14674.616899] LNet: Added LNI 192.168.204.119@tcp102 [8/256/0/180] [14675.452361] LNet: Added LNI 192.168.204.119@tcp103 [8/256/0/180] [14676.257274] LNet: Added LNI 192.168.204.119@tcp104 [8/256/0/180] [14677.122846] LNet: Added LNI 192.168.204.119@tcp105 [8/256/0/180] [14678.074503] LNet: Added LNI 192.168.204.119@tcp106 [8/256/0/180] [14678.851938] LNet: Added LNI 192.168.204.119@tcp107 [8/256/0/180] [14679.723825] LNet: Added LNI 192.168.204.119@tcp108 [8/256/0/180] [14680.686771] LNet: Added LNI 192.168.204.119@tcp109 [8/256/0/180] [14681.606082] LNet: Added LNI 192.168.204.119@tcp110 [8/256/0/180] [14682.836698] LNet: Added LNI 192.168.204.119@tcp111 [8/256/0/180] [14683.700460] LNet: Added LNI 192.168.204.119@tcp112 [8/256/0/180] [14684.701366] LNet: Added LNI 192.168.204.119@tcp113 [8/256/0/180] [14685.570696] LNet: Added LNI 192.168.204.119@tcp114 [8/256/0/180] [14686.655770] LNet: Added LNI 192.168.204.119@tcp115 [8/256/0/180] [14687.496856] LNet: Added LNI 192.168.204.119@tcp116 [8/256/0/180] [14688.199018] LNet: Added LNI 192.168.204.119@tcp117 [8/256/0/180] [14688.934303] LNet: Added LNI 192.168.204.119@tcp118 [8/256/0/180] [14689.707918] LNet: Added LNI 192.168.204.119@tcp119 [8/256/0/180] [14690.572194] LNet: Added LNI 192.168.204.119@tcp120 [8/256/0/180] [14691.405823] LNet: Added LNI 192.168.204.119@tcp121 [8/256/0/180] [14692.431273] LNet: Added LNI 192.168.204.119@tcp122 [8/256/0/180] [14693.497622] LNet: Added LNI 192.168.204.119@tcp123 [8/256/0/180] [14694.501929] LNet: Added LNI 192.168.204.119@tcp124 [8/256/0/180] [14695.469969] LNet: Added LNI 192.168.204.119@tcp125 [8/256/0/180] [14696.388691] LNet: Added LNI 192.168.204.119@tcp126 [8/256/0/180] [14697.317174] LNet: Added LNI 192.168.204.119@tcp127 [8/256/0/180] [14698.039133] LNet: Added LNI 192.168.204.119@tcp128 [8/256/0/180] [14699.026346] LNet: Added LNI 192.168.204.119@tcp129 [8/256/0/180] [14700.116603] LNet: Added LNI 192.168.204.119@tcp130 [8/256/0/180] [14700.967512] LNet: Added LNI 192.168.204.119@tcp131 [8/256/0/180] [14701.860595] LNet: Added LNI 192.168.204.119@tcp132 [8/256/0/180] [14702.718094] LNet: Added LNI 192.168.204.119@tcp133 [8/256/0/180] [14703.547671] LNet: Added LNI 192.168.204.119@tcp134 [8/256/0/180] [14704.251440] LNet: Added LNI 192.168.204.119@tcp135 [8/256/0/180] [14711.353730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14711.387352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14712.526777] Lustre: 463940:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.119@tcp: can't add NID 192.168.204.119@tcp130: rc = -75 [14712.605745] LNetError: 461387:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14712.620014] LNetError: Protocol error connecting to 192.168.204.119@tcp118 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14712.629962] LNetError: Skipped 4 previous similar messages [14712.725213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14712.755371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14712.883796] Lustre: lustre-MDT0000: new disk, initializing [14712.933244] LNet: 7 local NIs in recovery (showing 7): 192.168.204.119@tcp120, 192.168.204.119@tcp122, 192.168.204.119@tcp123, 192.168.204.119@tcp118, 192.168.204.119@tcp125, 192.168.204.119@tcp129, 192.168.204.119@tcp128 [14712.978236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14712.999125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14713.699176] LNetError: 463968:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14713.726628] LNetError: 463968:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14713.736202] LNetError: Protocol error connecting to 192.168.204.119@tcp126 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14713.763461] LNetError: Skipped 6 previous similar messages [14716.006555] LNetError: 463969:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14716.025476] LNetError: 463969:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14716.031220] LNetError: Protocol error connecting to 192.168.204.119@tcp120 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14716.042639] LNetError: Skipped 10 previous similar messages [14718.054851] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14720.103089] LNetError: 463960:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14720.121886] LNetError: 463960:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14720.134440] LNetError: Protocol error connecting to 192.168.204.119@tcp120 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14728.205445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14728.293551] LNetError: 463961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14728.312193] LNetError: Protocol error connecting to 192.168.204.119@tcp120 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14728.369925] Lustre: 463958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14728.498566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14728.509106] Lustre: Skipped 1 previous similar message [14728.639871] Lustre: lustre-MDT0001: new disk, initializing [14728.701172] Lustre: 464759:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.119@tcp: can't add NID 192.168.204.119@tcp130: rc = -75 [14728.712206] Lustre: 464759:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14728.879777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14728.913333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14728.928261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14733.509436] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14742.628691] LNetError: 463968:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14742.652407] LNetError: Protocol error connecting to 192.168.204.119@tcp126 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14743.074807] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14748.489829] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14754.651251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14754.945425] Lustre: lustre-OST0000: new disk, initializing [14754.949507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14754.955304] Lustre: 465933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14755.064270] Lustre: 465938:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.119@tcp: can't add NID 192.168.204.119@tcp130: rc = -75 [14755.073111] Lustre: 465938:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14755.115165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14759.016142] LNetError: 463965:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14759.034036] LNetError: 463965:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14759.043601] LNetError: Protocol error connecting to 192.168.204.119@tcp124 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14759.050391] LNetError: Skipped 13 previous similar messages [14760.818824] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14763.044143] Lustre: 466390:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.119@tcp: can't add NID 192.168.204.119@tcp130: rc = -75 [14763.056525] Lustre: 466390:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14763.137421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14763.151057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14763.287847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14769.413858] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14773.344205] LNet: 1 local NIs in recovery (showing 1): 192.168.204.119@tcp120 [14778.853266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14778.855749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14778.868491] Lustre: Skipped 1 previous similar message [14778.890141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14781.455331] Lustre: server umount lustre-OST0000 complete [14784.993537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14784.997878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14785.007671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14785.012597] Lustre: Skipped 1 previous similar message [14789.092280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14789.093825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14790.681111] Lustre: server umount lustre-MDT0000 complete [14792.802331] LNetError: 463960:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.119 [14792.820384] LNetError: 463960:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [14792.830369] LNetError: Protocol error connecting to 192.168.204.119@tcp128 at host 192.168.204.119:988: is it running a compatible version of Lustre? [14792.838473] LNetError: Skipped 20 previous similar messages [14793.919961] LustreError: 463950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861772 with bad export cookie 4036955434179999679 [14793.930092] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14793.941293] LustreError: 463950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14794.228852] Lustre: server umount lustre-MDT0001 complete [14804.726036] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [14807.406391] Key type lgssc unregistered [14807.799210] LNet: 467844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14807.813067] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14807.821836] LNet: Removed LNI 192.168.204.119@tcp [14808.866070] LNet: Removed LNI 192.168.204.119@tcp100 [14808.875638] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14808.883201] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14808.897871] LNet: Removed LNI 192.168.204.119@tcp101 [14809.952257] LNet: Removed LNI 192.168.204.119@tcp102 [14809.957850] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14809.963824] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14809.969982] LNet: Removed LNI 192.168.204.119@tcp103 [14810.976293] LNet: Removed LNI 192.168.204.119@tcp104 [14812.000190] LNet: Removed LNI 192.168.204.119@tcp105 [14812.008432] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14812.017561] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14812.026268] LNet: Removed LNI 192.168.204.119@tcp106 [14813.088304] LNet: Removed LNI 192.168.204.119@tcp107 [14814.112296] LNet: Removed LNI 192.168.204.119@tcp108 [14815.136257] LNet: Removed LNI 192.168.204.119@tcp109 [14816.160220] LNet: Removed LNI 192.168.204.119@tcp110 [14816.165468] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14816.174234] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14816.181526] LNet: Removed LNI 192.168.204.119@tcp111 [14817.249991] LNet: Removed LNI 192.168.204.119@tcp112 [14818.272140] LNet: Removed LNI 192.168.204.119@tcp113 [14819.296096] LNet: Removed LNI 192.168.204.119@tcp114 [14820.320146] LNet: Removed LNI 192.168.204.119@tcp115 [14821.347292] LNet: Removed LNI 192.168.204.119@tcp116 [14822.368512] LNet: Removed LNI 192.168.204.119@tcp117 [14823.392166] LNet: Removed LNI 192.168.204.119@tcp118 [14824.416812] LNet: Removed LNI 192.168.204.119@tcp119 [14824.423793] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14824.432706] LNetError: 467844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14824.439345] LNet: Removed LNI 192.168.204.119@tcp120 [14825.504187] LNet: Removed LNI 192.168.204.119@tcp121 [14826.528289] LNet: Removed LNI 192.168.204.119@tcp122 [14827.553876] LNet: Removed LNI 192.168.204.119@tcp123 [14828.576269] LNet: Removed LNI 192.168.204.119@tcp124 [14829.600185] LNet: Removed LNI 192.168.204.119@tcp125 [14830.624149] LNet: Removed LNI 192.168.204.119@tcp126 [14831.648886] LNet: Removed LNI 192.168.204.119@tcp127 [14832.674538] LNet: Removed LNI 192.168.204.119@tcp128 [14833.696152] LNet: Removed LNI 192.168.204.119@tcp129 [14833.700581] LNet: Removed LNI 192.168.204.119@tcp130 [14833.705905] LNet: Removed LNI 192.168.204.119@tcp131 [14833.709537] LNet: Removed LNI 192.168.204.119@tcp132 [14833.716931] LNet: Removed LNI 192.168.204.119@tcp133 [14833.718900] LNet: Removed LNI 192.168.204.119@tcp134 [14833.732828] LNet: Removed LNI 192.168.204.119@tcp135 [14834.716370] Key type .llcrypt unregistered [14834.720252] Key type ._llcrypt unregistered [14857.072066] Key type ._llcrypt registered [14857.074524] Key type .llcrypt registered [14857.176668] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [14871.203739] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [14872.319086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14872.337150] alg: No test for adler32 (adler32-zlib) [14873.401469] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [14873.582821] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [14875.280591] Key type lgssc registered [14876.404721] Lustre: Echo OBD driver; http://www.lustre.org/ [14908.063987] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:04:45 (1788861885) [14924.499992] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [14926.683215] Key type lgssc unregistered [14926.940766] LNet: 471923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14926.950418] LNetError: 471923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14926.966527] LNet: Removed LNI 192.168.204.119@tcp [14927.662074] Key type .llcrypt unregistered [14927.666638] Key type ._llcrypt unregistered [14942.961554] Key type ._llcrypt registered [14942.965680] Key type .llcrypt registered [14943.105790] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [14944.074218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14944.116558] alg: No test for adler32 (adler32-zlib) [14945.206075] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [14945.398360] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [14947.056558] Key type lgssc registered [14948.197977] Lustre: Echo OBD driver; http://www.lustre.org/ [14957.277501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14957.315048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14958.682600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14958.715870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14958.850459] Lustre: lustre-MDT0000: new disk, initializing [14958.971886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14959.007275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14964.256388] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14974.531437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14974.638575] Lustre: 473134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14974.686800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14974.690631] Lustre: Skipped 1 previous similar message [14974.749739] Lustre: lustre-MDT0001: new disk, initializing [14974.823761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14974.866566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14974.883333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14979.584809] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [14989.088444] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14994.745803] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15001.513887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15001.941575] Lustre: lustre-OST0000: new disk, initializing [15001.952871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15001.970055] Lustre: 475104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15002.088553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15007.310899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15007.321958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15007.418361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15007.891972] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15016.393515] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15020.005521] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing lnet_if_list [15024.280755] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15024.337101] LNet: Added LNI 192.168.204.119@tcp10 [8/256/0/180] [15029.276465] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15029.309788] LNet: Added LNI 192.168.204.119@tcp11 [8/256/0/180] [15032.947708] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15032.975035] LNet: Added LNI 192.168.204.119@tcp12 [8/256/0/180] [15036.308683] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15036.344672] LNet: Added LNI 192.168.204.119@tcp13 [8/256/0/180] [15039.778561] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15039.813769] LNet: Added LNI 192.168.204.119@tcp14 [8/256/0/180] [15043.812548] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15043.848652] LNet: Added LNI 192.168.204.119@tcp15 [8/256/0/180] [15047.768615] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15047.794469] LNet: Added LNI 192.168.204.119@tcp16 [8/256/0/180] [15051.697970] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15051.720927] LNet: Added LNI 192.168.204.119@tcp17 [8/256/0/180] [15055.687904] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15055.708627] LNet: Added LNI 192.168.204.119@tcp18 [8/256/0/180] [15060.182353] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15060.212421] LNet: Added LNI 192.168.204.119@tcp19 [8/256/0/180] [15064.927474] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15064.955048] LNet: Added LNI 192.168.204.119@tcp20 [8/256/0/180] [15069.411754] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15069.436137] LNet: Added LNI 192.168.204.119@tcp21 [8/256/0/180] [15074.211174] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15074.258261] LNet: Added LNI 192.168.204.119@tcp22 [8/256/0/180] [15078.649905] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15078.670959] LNet: Added LNI 192.168.204.119@tcp23 [8/256/0/180] [15083.168888] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15083.196597] LNet: Added LNI 192.168.204.119@tcp24 [8/256/0/180] [15088.581046] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15088.601772] LNet: Added LNI 192.168.204.119@tcp25 [8/256/0/180] [15093.767874] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15093.790521] LNet: Added LNI 192.168.204.119@tcp26 [8/256/0/180] [15098.737762] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15098.761884] LNet: Added LNI 192.168.204.119@tcp27 [8/256/0/180] [15103.852881] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15103.890745] LNet: Added LNI 192.168.204.119@tcp28 [8/256/0/180] [15109.474742] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15109.496496] LNet: Added LNI 192.168.204.119@tcp29 [8/256/0/180] [15115.001735] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15115.047847] LNet: Added LNI 192.168.204.119@tcp30 [8/256/0/180] [15120.539258] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15120.574949] LNet: Added LNI 192.168.204.119@tcp31 [8/256/0/180] [15126.619579] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15126.647451] LNet: Added LNI 192.168.204.119@tcp32 [8/256/0/180] [15132.948094] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15132.992758] LNet: Added LNI 192.168.204.119@tcp33 [8/256/0/180] [15139.587822] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15139.615330] LNet: Added LNI 192.168.204.119@tcp34 [8/256/0/180] [15146.475805] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15146.509492] LNet: Added LNI 192.168.204.119@tcp35 [8/256/0/180] [15152.882747] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15152.917673] LNet: Added LNI 192.168.204.119@tcp36 [8/256/0/180] [15160.296503] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15160.339528] LNet: Added LNI 192.168.204.119@tcp37 [8/256/0/180] [15167.261117] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15167.311976] LNet: Added LNI 192.168.204.119@tcp38 [8/256/0/180] [15174.879897] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15174.918343] LNet: Added LNI 192.168.204.119@tcp39 [8/256/0/180] [15182.151233] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15182.180786] LNet: Added LNI 192.168.204.119@tcp40 [8/256/0/180] [15188.587697] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15188.625411] LNet: Added LNI 192.168.204.119@tcp41 [8/256/0/180] [15195.752377] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15195.799462] LNet: Added LNI 192.168.204.119@tcp42 [8/256/0/180] [15203.629553] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15203.650870] LNet: Added LNI 192.168.204.119@tcp43 [8/256/0/180] [15212.007872] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15212.034401] LNet: Added LNI 192.168.204.119@tcp44 [8/256/0/180] [15219.618291] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15219.639599] LNet: Added LNI 192.168.204.119@tcp45 [8/256/0/180] [15228.168830] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15228.191289] LNet: Added LNI 192.168.204.119@tcp46 [8/256/0/180] [15236.725676] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15236.755738] LNet: Added LNI 192.168.204.119@tcp47 [8/256/0/180] [15245.599108] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15245.697985] LNet: Added LNI 192.168.204.119@tcp48 [8/256/0/180] [15254.783103] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15254.820034] LNet: Added LNI 192.168.204.119@tcp49 [8/256/0/180] [15263.392340] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15263.430427] LNet: Added LNI 192.168.204.119@tcp50 [8/256/0/180] [15282.147577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15282.158278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15282.196767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15284.206495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15284.210090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15284.223770] Lustre: Skipped 1 previous similar message [15284.235792] Lustre: Skipped 2 previous similar messages [15288.250923] Lustre: server umount lustre-MDT0000 complete [15291.985723] LustreError: 473126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862270 with bad export cookie 13325032530234470761 [15291.991689] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15292.008057] LustreError: 473126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15292.435209] Lustre: server umount lustre-MDT0001 complete [15310.816415] Lustre: 472408:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862273/real 1788862273] req@00000000a07bca00 x1875757680979200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15310.875146] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15311.149942] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [15312.874086] Lustre: 472405:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862275/real 1788862275] req@0000000019151b9c x1875757680979328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15316.006071] Lustre: 472405:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862278/real 1788862278] req@0000000079b1feea x1875757680979456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15319.504313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15323.104374] Lustre: 472408:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862285/real 1788862285] req@000000004267e2b3 x1875757680979840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15323.153192] Lustre: 472408:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15336.802043] LustreError: 475098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15336.825867] LustreError: 475098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15336.873785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15339.950627] LustreError: 475099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15339.955217] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15340.966182] LustreError: 475098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15341.111177] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15346.156000] LustreError: 475099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15350.102528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15350.507313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15355.105756] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15355.646759] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15362.863526] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15368.778898] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15388.137759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15388.147048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15388.157817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15391.714687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15391.717377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15391.744620] Lustre: Skipped 1 previous similar message [15394.582152] Lustre: server umount lustre-OST0000 complete [15401.957848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15401.959864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15401.972452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15401.998657] Lustre: Skipped 2 previous similar messages [15402.013971] Lustre: Skipped 1 previous similar message [15404.937154] Lustre: server umount lustre-MDT0000 complete [15407.085524] LustreError: 499345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15407.105029] LustreError: 499345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15409.013185] LustreError: 473127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862387 with bad export cookie 13325032530234473295 [15409.023283] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15409.033920] LustreError: 473127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15409.306696] Lustre: server umount lustre-MDT0001 complete [15419.242945] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [15421.789924] Key type lgssc unregistered [15422.122323] LNet: 502142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15422.133231] LNetError: 502142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15423.200446] LNet: Removed LNI 192.168.204.119@tcp [15423.205339] LNetError: 502142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15423.214488] LNet: Removed LNI 192.168.204.119@tcp10 [15423.222017] LNet: Removed LNI 192.168.204.119@tcp11 [15423.230506] LNet: Removed LNI 192.168.204.119@tcp12 [15423.233888] LNet: Removed LNI 192.168.204.119@tcp13 [15423.237961] LNet: Removed LNI 192.168.204.119@tcp14 [15423.242744] LNet: Removed LNI 192.168.204.119@tcp15 [15423.248622] LNet: Removed LNI 192.168.204.119@tcp16 [15423.252523] LNet: Removed LNI 192.168.204.119@tcp17 [15423.255775] LNet: Removed LNI 192.168.204.119@tcp18 [15423.259838] LNet: Removed LNI 192.168.204.119@tcp19 [15423.264586] LNet: Removed LNI 192.168.204.119@tcp20 [15423.267259] LNet: Removed LNI 192.168.204.119@tcp21 [15423.270852] LNet: Removed LNI 192.168.204.119@tcp22 [15423.274589] LNet: Removed LNI 192.168.204.119@tcp23 [15423.280549] LNet: Removed LNI 192.168.204.119@tcp24 [15423.286043] LNet: Removed LNI 192.168.204.119@tcp25 [15423.294216] LNet: Removed LNI 192.168.204.119@tcp26 [15423.301409] LNet: Removed LNI 192.168.204.119@tcp27 [15423.304556] LNet: Removed LNI 192.168.204.119@tcp28 [15423.307746] LNet: Removed LNI 192.168.204.119@tcp29 [15423.312405] LNet: Removed LNI 192.168.204.119@tcp30 [15423.316836] LNet: Removed LNI 192.168.204.119@tcp31 [15423.319238] LNet: Removed LNI 192.168.204.119@tcp32 [15423.321573] LNet: Removed LNI 192.168.204.119@tcp33 [15423.324570] LNet: Removed LNI 192.168.204.119@tcp34 [15423.326774] LNet: Removed LNI 192.168.204.119@tcp35 [15423.328898] LNet: Removed LNI 192.168.204.119@tcp36 [15423.334128] LNet: Removed LNI 192.168.204.119@tcp37 [15423.337902] LNet: Removed LNI 192.168.204.119@tcp38 [15423.347937] LNet: Removed LNI 192.168.204.119@tcp39 [15423.350667] LNet: Removed LNI 192.168.204.119@tcp40 [15423.353836] LNet: Removed LNI 192.168.204.119@tcp41 [15423.357074] LNet: Removed LNI 192.168.204.119@tcp42 [15423.367831] LNet: Removed LNI 192.168.204.119@tcp43 [15423.372719] LNet: Removed LNI 192.168.204.119@tcp44 [15423.376869] LNet: Removed LNI 192.168.204.119@tcp45 [15423.379949] LNet: Removed LNI 192.168.204.119@tcp46 [15423.383842] LNet: Removed LNI 192.168.204.119@tcp47 [15423.386466] LNet: Removed LNI 192.168.204.119@tcp48 [15423.388946] LNet: Removed LNI 192.168.204.119@tcp49 [15423.401627] LNet: Removed LNI 192.168.204.119@tcp50 [15424.263669] Key type .llcrypt unregistered [15424.267554] Key type ._llcrypt unregistered [15445.019889] Key type ._llcrypt registered [15445.021845] Key type .llcrypt registered [15445.137344] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [15461.879166] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [15463.078689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15463.297571] alg: No test for adler32 (adler32-zlib) [15464.403596] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [15464.711949] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [15466.457292] Key type lgssc registered [15468.080951] Lustre: Echo OBD driver; http://www.lustre.org/ [15508.076486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15508.106083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15509.346704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15509.374989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15509.467495] Lustre: lustre-MDT0000: new disk, initializing [15509.586079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15509.606260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15513.794915] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15523.321927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15523.456605] Lustre: 505337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15523.538247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15523.544358] Lustre: Skipped 1 previous similar message [15523.621435] Lustre: lustre-MDT0001: new disk, initializing [15523.710201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15523.747203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15523.763141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15528.202732] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15537.244431] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15542.505972] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15550.098115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15550.231698] Lustre: *** cfs_fail_loc=90f, val=0*** [15550.298507] Lustre: lustre-OST0000: new disk, initializing [15550.302831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15550.308836] Lustre: 507355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15550.392371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15556.463190] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15558.237898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15558.252762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15558.343625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15566.066799] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15573.987432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15574.002920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15574.010977] LustreError: Skipped 1 previous similar message [15574.035968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15574.048296] Lustre: Skipped 1 previous similar message [15579.109949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15579.124015] Lustre: Skipped 2 previous similar messages [15579.796658] Lustre: server umount lustre-OST0000 complete [15584.242672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15584.245975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15584.264517] Lustre: Skipped 1 previous similar message [15584.284153] Lustre: Skipped 1 previous similar message [15589.345754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15589.363229] Lustre: Skipped 1 previous similar message [15589.904277] Lustre: server umount lustre-MDT0000 complete [15594.314743] LustreError: 505328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862572 with bad export cookie 13807286190500297701 [15594.319609] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15594.333490] LustreError: 505328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15594.832907] Lustre: server umount lustre-MDT0001 complete [15605.907687] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [15609.523223] Key type lgssc unregistered [15609.900904] LNet: 509260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15609.930519] LNetError: 509260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15609.952625] LNet: Removed LNI 192.168.204.119@tcp [15610.793888] Key type .llcrypt unregistered [15610.798503] Key type ._llcrypt unregistered [15629.743643] Key type ._llcrypt registered [15629.756129] Key type .llcrypt registered [15629.878659] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [15631.394471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15631.413218] alg: No test for adler32 (adler32-zlib) [15632.640190] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [15633.020909] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [15634.696468] Key type lgssc registered [15636.002857] Lustre: Echo OBD driver; http://www.lustre.org/ [15645.316480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15645.332971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15646.999264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15652.715769] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15656.438338] LustreError: 510526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15656.478235] LustreError: 510526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15661.538960] LustreError: 510527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15662.571493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15663.429533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15669.592381] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15678.367459] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15684.357830] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15692.402118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15692.967216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15700.875663] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15709.096959] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15730.919921] loop0: detected capacity change from 0 to 204800000 [15731.202953] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15731.234922] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15731.244932] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15737.867128] loop0: detected capacity change from 0 to 204800000 [15738.047555] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15738.058778] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15738.067793] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15743.035173] loop0: detected capacity change from 0 to 204800000 [15748.946956] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15749.177584] Lustre: fs15246-OST0000: new disk, initializing [15749.186745] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15749.195650] Lustre: 514144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15749.276243] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15755.949277] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15766.723959] loop1: detected capacity change from 0 to 204800000 [15772.480513] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15772.613193] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15772.679676] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15772.751336] Lustre: fs15246-MDT0000: new disk, initializing [15772.996830] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15773.044938] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15773.225218] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15774.243068] Lustre: 515267:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000080534b3a x1875758401243136/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788862768 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15778.293777] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15783.418298] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15783.433995] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15783.466718] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15805.432583] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15805.454591] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15807.693151] Lustre: server umount fs15246-MDT0000 complete [15810.541901] LustreError: 510521:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15815.658807] LustreError: 510521:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15818.289408] Lustre: server umount fs15246-OST0000 complete [15831.014157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15831.019574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15831.026507] Lustre: Skipped 1 previous similar message [15831.050086] Lustre: Skipped 1 previous similar message [15836.136442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15836.144549] Lustre: Skipped 1 previous similar message [15836.599350] Lustre: server umount lustre-OST0000 complete [15841.255185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15841.265858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15841.271606] Lustre: Skipped 1 previous similar message [15846.152808] Lustre: server umount lustre-MDT0000 complete [15846.375863] LustreError: 510521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15846.399903] LustreError: 510521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15849.429621] LustreError: 510506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862828 with bad export cookie 1633974225592982318 [15849.431921] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15849.447544] LustreError: 510506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15849.736978] Lustre: server umount lustre-MDT0001 complete [15859.701635] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [15861.930170] Key type lgssc unregistered [15862.249830] LNet: 518361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15862.260754] LNetError: 518361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15862.295621] LNet: Removed LNI 192.168.204.119@tcp [15863.084437] Key type .llcrypt unregistered [15863.087386] Key type ._llcrypt unregistered [15881.402729] Key type ._llcrypt registered [15881.405073] Key type .llcrypt registered [15881.520331] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [15882.383241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15882.416626] alg: No test for adler32 (adler32-zlib) [15883.569543] Lustre: Lustre: Build Version: 2.17.58_39_g86006df [15883.833398] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [15885.560216] Key type lgssc registered [15886.495894] Lustre: Echo OBD driver; http://www.lustre.org/ [15921.937053] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [15930.755692] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [15970.252288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15970.269078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15971.516900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15971.541283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15971.608510] Lustre: lustre-MDT0000: new disk, initializing [15971.706698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15971.728687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15975.811409] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15985.533909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15985.676894] Lustre: 522822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15985.741891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15985.748573] Lustre: Skipped 1 previous similar message [15985.823882] Lustre: lustre-MDT0001: new disk, initializing [15985.909568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15985.952763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15985.961624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15990.204976] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [15999.361548] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16004.444761] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16010.167448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16010.406324] Lustre: lustre-OST0000: new disk, initializing [16010.409584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16010.414643] Lustre: 524795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16010.495434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16016.776399] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [16019.029787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16019.037514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16019.160634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16026.488108] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16033.256924] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16033.582988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16038.256073] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16038.491338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16045.027082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16045.027113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16045.032260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16045.039594] LustreError: Skipped 1 previous similar message [16045.052880] Lustre: Skipped 1 previous similar message [16046.731799] Lustre: server umount lustre-OST0000 complete [16052.706365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16052.709567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16052.724297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16052.732910] Lustre: Skipped 1 previous similar message [16055.267052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16055.268207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16055.279372] Lustre: Skipped 1 previous similar message [16056.869403] Lustre: server umount lustre-MDT0000 complete [16061.198910] LustreError: 522817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863039 with bad export cookie 979442581286623384 [16061.211532] LustreError: 522817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16061.213400] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16061.603167] Lustre: server umount lustre-MDT0001 complete [16088.120800] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [16097.469575] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [16130.220666] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15780 sec ========= 06:25:07 (1788863107) [16131.783781] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:25:08 (1788863108) === [16135.670445] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:25:12 (1788863112) === [16177.404282] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [16180.484071] Key type lgssc unregistered [16180.862947] LNet: 531437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16180.875794] LNetError: 531437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16180.898222] LNet: Removed LNI 192.168.204.119@tcp [16181.706040] Key type .llcrypt unregistered [16181.708391] Key type ._llcrypt unregistered