[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 733377472 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003341] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007646] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010154] LSM: Security Framework initializing [ 0.011076] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021780] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030145] *** VALIDATE tmpfs *** [ 0.032377] *** VALIDATE proc *** [ 0.033267] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036159] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039045] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043441] debug: unmapping init [mem 0xffffffff91659000-0xffffffff91660fff] [ 0.046272] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053034] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055404] rcu: Hierarchical SRCU implementation. [ 0.057665] smp: Bringing up secondary CPUs ... [ 0.058987] x86: Booting SMP configuration: [ 0.059035] .... node #0, CPUs: #1 #2 #3 [ 0.070021] smp: Brought up 1 node, 4 CPUs [ 0.072013] smpboot: Max logical packages: 1 [ 0.073016] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.106760] node 0 deferred pages initialised in 31ms [ 0.112214] devtmpfs: initialized [ 0.114275] x86/mm: Memory block size: 128MB [ 0.117208] gcov: version magic: 0x41383552 [ 0.120174] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121176] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.122299] pinctrl core: initialized pinctrl subsystem [ 0.123365] [ 0.123991] ************************************************************* [ 0.124018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125016] ** ** [ 0.126018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.127014] ** ** [ 0.128016] ** This means that this kernel is built to expose internal ** [ 0.129017] ** IOMMU data structures, which may compromise security on ** [ 0.130016] ** your system. ** [ 0.131019] ** ** [ 0.132018] ** If you see this message and you are not debugging the ** [ 0.133019] ** kernel, report this immediately to your vendor! ** [ 0.134019] ** ** [ 0.135016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136016] ************************************************************* [ 0.137878] NET: Registered protocol family 16 [ 0.138572] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.139063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.140081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.141797] cpuidle: using governor menu [ 0.144749] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.149085] PCI: Using configuration type 1 for base access [ 0.152141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.182320] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227164] cryptd: max_cpu_qlen set to 1000 [ 0.229275] ACPI: Added _OSI(Module Device) [ 0.230011] ACPI: Added _OSI(Processor Device) [ 0.232010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239439] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247617] ACPI: Interpreter enabled [ 0.250200] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252014] ACPI: Using IOAPIC for interrupt routing [ 0.254242] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.259616] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273825] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293551] acpiphp: Slot [2] registered [ 0.295148] acpiphp: Slot [5] registered [ 0.296136] acpiphp: Slot [6] registered [ 0.298125] acpiphp: Slot [7] registered [ 0.299114] acpiphp: Slot [8] registered [ 0.300098] acpiphp: Slot [9] registered [ 0.302206] acpiphp: Slot [10] registered [ 0.303130] acpiphp: Slot [3] registered [ 0.305104] acpiphp: Slot [4] registered [ 0.306088] acpiphp: Slot [11] registered [ 0.307080] acpiphp: Slot [12] registered [ 0.309088] acpiphp: Slot [13] registered [ 0.310087] acpiphp: Slot [14] registered [ 0.311083] acpiphp: Slot [15] registered [ 0.312191] acpiphp: Slot [16] registered [ 0.313000] acpiphp: Slot [17] registered [ 0.315090] acpiphp: Slot [18] registered [ 0.317090] acpiphp: Slot [19] registered [ 0.318088] acpiphp: Slot [20] registered [ 0.320084] acpiphp: Slot [21] registered [ 0.321109] acpiphp: Slot [22] registered [ 0.322108] acpiphp: Slot [23] registered [ 0.324097] acpiphp: Slot [24] registered [ 0.326115] acpiphp: Slot [25] registered [ 0.327099] acpiphp: Slot [26] registered [ 0.328190] acpiphp: Slot [27] registered [ 0.329000] acpiphp: Slot [28] registered [ 0.329000] acpiphp: Slot [29] registered [ 0.331082] acpiphp: Slot [30] registered [ 0.333095] acpiphp: Slot [31] registered [ 0.334057] PCI host bridge to bus 0000:00 [ 0.335023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348087] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350319] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366840] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372884] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.380016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382656] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386023] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.414011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423848] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.468018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.483735] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.535472] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.557018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.620038] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626433] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.633000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.644020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682060] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.692020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.703023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.725750] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.734018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.738021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.757029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.771037] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.775836] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.778500] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.781388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.784228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790075] iommu: Default domain type: Passthrough [ 0.791000] SCSI subsystem initialized [ 0.792169] ACPI: bus type USB registered [ 0.794142] usbcore: registered new interface driver usbfs [ 0.796196] usbcore: registered new interface driver hub [ 0.799113] usbcore: registered new device driver usb [ 0.801207] pps_core: LinuxPPS API ver. 1 registered [ 0.803014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.807099] PTP clock support registered [ 0.810092] EDAC MC: Ver: 3.0.0 [ 0.812131] PCI: Using ACPI for IRQ routing [ 0.814069] NetLabel: Initializing [ 0.815010] NetLabel: domain hash size = 128 [ 0.817012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820155] NetLabel: unlabeled traffic allowed by default [ 0.822478] vgaarb: loaded [ 0.824582] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.827032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.834345] clocksource: Switched to clocksource kvm-clock [ 0.953806] VFS: Disk quotas dquot_6.6.0 [ 0.955471] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.957771] *** VALIDATE ramfs *** [ 0.958888] *** VALIDATE hugetlbfs *** [ 0.960363] pnp: PnP ACPI init [ 0.962748] pnp: PnP ACPI: found 6 devices [ 0.977904] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980934] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.983234] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985538] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.987779] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.990140] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.993366] NET: Registered protocol family 2 [ 0.996111] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.001585] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.005920] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.012215] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.016928] TCP: Hash tables configured (established 65536 bind 65536) [ 1.020814] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.024567] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027936] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031472] NET: Registered protocol family 1 [ 1.034286] RPC: Registered named UNIX socket transport module. [ 1.037401] RPC: Registered udp transport module. [ 1.039657] RPC: Registered tcp transport module. [ 1.042139] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.045540] NET: Registered protocol family 44 [ 1.047758] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.050758] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.053613] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.056178] PCI: CLS 0 bytes, default 64 [ 1.058072] Unpacking initramfs... [ 2.705177] debug: unmapping init [mem 0xffff934d7cc54000-0xffff934d7ffbffff] [ 2.710379] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.713388] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.718464] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 3.400476] Initialise system trusted keyrings [ 3.402471] Key type blacklist registered [ 3.404608] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.414946] zbud: loaded [ 3.418140] *** VALIDATE nfs *** [ 3.419539] *** VALIDATE nfs4 *** [ 3.421532] pstore: using deflate compression [ 3.425694] Platform Keyring initialized [ 3.574601] NET: Registered protocol family 38 [ 3.577923] Key type asymmetric registered [ 3.580570] Asymmetric key parser 'x509' registered [ 3.583813] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.589365] io scheduler mq-deadline registered [ 3.591533] io scheduler kyber registered [ 3.594895] io scheduler bfq registered [ 3.597139] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.601737] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.607893] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.612523] ACPI: Power Button [PWRF] [ 3.621572] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.632838] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.652613] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.662569] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.686729] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.721362] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.751506] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.756670] Non-volatile memory driver v1.3 [ 3.758053] Linux agpgart interface v0.103 [ 3.791733] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 3.794643] vda: detected capacity change from 0 to 69050368 [ 3.837642] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.841089] vdb: detected capacity change from 0 to 1073741824 [ 3.872927] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.877507] vdc: detected capacity change from 0 to 2621440000 [ 3.915294] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.918068] vdd: detected capacity change from 0 to 2621440000 [ 3.955513] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.961682] vde: detected capacity change from 0 to 4294967296 [ 3.990528] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.993307] vdf: detected capacity change from 0 to 4294967296 [ 4.003575] libphy: Fixed MDIO Bus: probed [ 4.009711] usbcore: registered new interface driver usbserial_generic [ 4.012112] usbserial: USB Serial support registered for generic [ 4.016735] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.021845] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.023798] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.026150] mousedev: PS/2 mouse device common for all mice [ 4.029223] rtc_cmos 00:05: RTC can wake from S4 [ 4.036192] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.037717] rtc_cmos 00:05: registered as rtc0 [ 4.042555] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.044908] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.046921] intel_pstate: CPU model not supported [ 4.077655] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.078438] hid: raw HID events driver (C) Jiri Kosina [ 4.085057] usbcore: registered new interface driver usbhid [ 4.086927] usbhid: USB HID core driver [ 4.089656] drop_monitor: Initializing network drop monitor service [ 4.092916] Initializing XFRM netlink socket [ 4.095146] NET: Registered protocol family 10 [ 4.100795] Segment Routing with IPv6 [ 4.102203] NET: Registered protocol family 17 [ 4.104316] mpls_gso: MPLS GSO support [ 4.110301] RAS: Correctable Errors collector initialized. [ 4.112306] AVX version of gcm_enc/dec engaged. [ 4.113742] AES CTR mode by8 optimization enabled [ 4.231968] sched_clock: Marking stable (4231951032, 0)->(5474976830, -1243025798) [ 4.235880] registered taskstats version 1 [ 4.237745] Loading compiled-in X.509 certificates [ 4.239986] zswap: loaded using pool lzo/zbud [ 4.270894] Key type big_key registered [ 4.285454] Key type encrypted registered [ 4.288512] ima: No TPM chip found, activating TPM-bypass! [ 4.290695] ima: Allocated hash algorithm: sha1 [ 4.292597] ima: No architecture policies found [ 4.294657] evm: Initialising EVM extended attributes: [ 4.296610] evm: security.selinux [ 4.297910] evm: security.ima [ 4.298970] evm: security.capability [ 4.300567] evm: HMAC attrs: 0x1 [ 4.304835] rtc_cmos 00:05: setting system clock to 2026-06-01 15:13:45 UTC (1780326825) [ 4.311970] debug: unmapping init [mem 0xffffffff92603000-0xffffffff927fffff] [ 4.315267] debug: unmapping init [mem 0xffffffff91382000-0xffffffff91658fff] [ 4.324234] Write protecting the kernel read-only data: 28672k [ 4.328466] debug: unmapping init [mem 0xffffffff8fa03000-0xffffffff8fbfffff] [ 4.331678] debug: unmapping init [mem 0xffffffff90314000-0xffffffff903fffff] [ 4.393480] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.405344] systemd[1]: Detected virtualization kvm. [ 4.407645] systemd[1]: Detected architecture x86-64. [ 4.409442] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.443524] systemd[1]: No hostname configured. [ 4.445949] systemd[1]: Set hostname to . [ 4.449110] random: systemd: uninitialized urandom read (16 bytes read) [ 4.452920] systemd[1]: Initializing machine ID from random generator. [ 4.781289] random: systemd: uninitialized urandom read (16 bytes read) [ 4.784221] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.814687] random: systemd: uninitialized urandom read (16 bytes read) [ 4.819709] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.839786] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.085693] device-mapper: uevent: version 1.0.3 [ 7.090490] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.716643] virtio_net virtio0 ens2: renamed from eth0 [ 8.757717] random: fast init done [ 9.233081] scsi host0: ata_piix [ 9.281660] scsi host1: ata_piix [ 9.287105] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.290139] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.900801] random: crng init done [ 14.902990] random: 7 urandom warning(s) missed due to ratelimiting [ 17.077584] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.459602] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.919567] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.332781] SELinux: Disabled at runtime. [ 21.397987] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.408416] systemd[1]: Detected virtualization kvm. [ 21.410799] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.357659] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.362475] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.373797] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.378781] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.383434] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.427782] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.442449] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 22.920717] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.713345] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.829782] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.272305] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.319695] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.478720] EDAC sbridge: Ver: 1.1.2 [ 29.617726] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.040970] NFS: Registering the id_resolver key type [ 30.043579] Key type id_resolver registered [ 30.045710] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Hostname Service... Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 70.065574] libcfs: loading out-of-tree module taints kernel. [ 70.084142] Key type ._llcrypt registered [ 70.085171] Key type .llcrypt registered [ 70.132321] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 79.683573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 80.468367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.478610] alg: No test for adler32 (adler32-zlib) [ 81.580192] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 82.048708] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 83.719519] Key type lgssc registered [ 84.719847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.139890] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.501785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 125.502577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.543375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.688730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 126.715347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.771785] Lustre: lustre-MDT0000: new disk, initializing [ 126.851172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.872874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.390499] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.120573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.175029] Lustre: 6476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 137.201952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 137.207446] Lustre: Skipped 1 previous similar message [ 137.263908] Lustre: lustre-MDT0001: new disk, initializing [ 137.296223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.311468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 137.316974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 137.348649] LustreError: 7427:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 137.357774] LustreError: 7427:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 137.366030] LustreError: 7427:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 139.375411] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.322672] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 147.732893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 147.864942] Lustre: lustre-OST0000: new disk, initializing [ 147.867356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 147.871596] Lustre: 8384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 147.872946] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 147.898666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.716162] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.078294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 153.084566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 153.107349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 158.272997] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 158.334936] Lustre: lustre-OST0001: new disk, initializing [ 158.337827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 158.341848] Lustre: 9438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 158.373993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.010520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 160.017234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 160.047479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 161.211209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.111479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 172.712872] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 176.129940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 178.232441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 180.769207] Lustre: DEBUG MARKER: Client: 2.17.53.33 [ 181.958596] Lustre: DEBUG MARKER: MDS: 2.17.53.33 [ 183.092399] Lustre: DEBUG MARKER: OSS: 2.17.53.33 [ 183.965684] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 11:16:44 EDT 2026 [ 193.392543] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 194.193204] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 208.863728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 208.868188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.877492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 211.423758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 211.430028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.440322] Lustre: Skipped 2 previous similar messages [ 213.324857] Lustre: server umount lustre-MDT0000 complete [ 215.266429] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327036 with bad export cookie 9936544131358893297 [ 215.266730] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 215.271051] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 216.543950] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 216.544489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 216.555798] Lustre: Skipped 1 previous similar message [ 216.560527] Lustre: Skipped 4 previous similar messages [ 220.577810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 220.581540] Lustre: Skipped 1 previous similar message [ 221.535339] Lustre: server umount lustre-MDT0001 complete [ 223.898291] Lustre: server umount lustre-OST0000 complete [ 226.223094] Lustre: server umount lustre-OST0001 complete [ 229.769220] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 234.914593] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 261.913060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.095528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 262.121888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 262.195560] Lustre: lustre-MDT0000: new disk, initializing [ 262.276021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.289038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 265.348850] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 272.659373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.717912] Lustre: 14940:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 272.753256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 272.759940] Lustre: Skipped 1 previous similar message [ 272.816470] Lustre: lustre-MDT0001: new disk, initializing [ 272.874267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.901160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 272.905487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.439323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 280.469028] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 282.770686] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 287.002641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.171259] Lustre: 16686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 288.544190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 288.613434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 290.974734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 295.886304] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 298.393275] hrtimer: interrupt took 3303535 ns [ 299.042412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 299.209244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 302.266298] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 302.406383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 304.112509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.118256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.133587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.154315] Lustre: Skipped 1 previous similar message [ 310.010017] Lustre: server umount lustre-OST0000 complete [ 313.825171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.835684] LustreError: Skipped 1 previous similar message [ 313.838583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.847258] Lustre: Skipped 1 previous similar message [ 318.638323] Lustre: server umount lustre-MDT0000 complete [ 319.457802] LustreError: 14952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.469139] LustreError: 14952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 321.295760] LustreError: 15723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327142 with bad export cookie 9936544131358894900 [ 321.297937] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.304658] LustreError: 15723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 321.522145] Lustre: server umount lustre-MDT0001 complete [ 324.153971] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:19:04 (1780327144) [ 326.249904] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 327.542629] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:19:07 (1780327147) [ 332.171133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.540354] LustreError: 18945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.600338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.606526] Lustre: Skipped 1 previous similar message [ 335.378107] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 337.896330] LustreError: 18946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.986738] LustreError: 18945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.819872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.648852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 351.224379] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 354.433914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 359.951413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 364.976958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 369.048676] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 388.067198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 388.075067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.080109] Lustre: Skipped 1 previous similar message [ 388.084721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 388.089923] Lustre: Skipped 6 previous similar messages [ 392.103791] Lustre: server umount lustre-OST0000 complete [ 401.510345] Lustre: server umount lustre-MDT0000 complete [ 403.426385] LustreError: 18941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.441344] LustreError: 18941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 404.425991] LustreError: 21414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327225 with bad export cookie 9936544131358895824 [ 404.434885] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.446990] LustreError: 21414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 404.701328] Lustre: server umount lustre-MDT0001 complete [ 412.692940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 414.871280] Key type lgssc unregistered [ 415.062704] LNet: 22148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.067141] LNetError: 22148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 415.075737] LNet: Removed LNI 192.168.204.146@tcp [ 415.658150] Key type .llcrypt unregistered [ 415.659841] Key type ._llcrypt unregistered [ 429.457774] Key type ._llcrypt registered [ 429.459365] Key type .llcrypt registered [ 429.522216] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 430.057981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 430.076946] alg: No test for adler32 (adler32-zlib) [ 431.019618] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 431.153975] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 432.791188] Key type lgssc registered [ 433.447763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 438.255155] loop: module loaded [ 438.290480] loop0: detected capacity change from 0 to 204800000 [ 438.450218] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 438.458239] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 438.665831] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.257984] loop0: detected capacity change from 0 to 204800000 [ 442.359344] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.367467] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.376381] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 445.669570] loop0: detected capacity change from 0 to 204800000 [ 449.230957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 449.246065] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.415157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.448596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.516863] Lustre: lustre-MDT0000: new disk, initializing [ 450.573073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.586440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.625457] Lustre: 24316:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 263, rollback = 2 [ 450.638197] Lustre: 24316:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 450.651104] Lustre: 24316:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/263/0 [ 450.662236] Lustre: 24316:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 450.666708] Lustre: 24316:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 450.673430] Lustre: 24316:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 452.988992] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 459.020428] loop1: detected capacity change from 0 to 204800000 [ 462.458595] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.622823] Lustre: lustre-OST07c6: new disk, initializing [ 462.626723] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 462.630043] Lustre: Skipped 1 previous similar message [ 462.633844] Lustre: 25277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.678898] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 464.157419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 464.165296] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 464.187588] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 465.788588] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 468.605604] Lustre: 25408:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 468.609922] Lustre: 25408:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 468.613060] Lustre: 25408:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 468.616362] Lustre: 25408:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 468.622112] Lustre: 25408:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 468.627877] Lustre: 25408:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 488.414264] Lustre: 24282:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 488.422523] Lustre: 24282:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 488.428926] Lustre: 24282:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 488.438241] Lustre: 24282:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 488.444109] Lustre: 24282:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 488.447567] Lustre: 24282:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 492.371605] Lustre: 24282:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 492.377731] Lustre: 24282:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 492.382901] Lustre: 24282:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 492.389458] Lustre: 24282:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.394276] Lustre: 24282:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 492.398539] Lustre: 24282:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.405105] Lustre: 24282:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 492.410308] Lustre: 24282:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.415189] Lustre: 24282:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 492.420599] Lustre: 24282:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.426970] Lustre: 24282:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 492.430530] Lustre: 24282:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 499.320942] Lustre: 24283:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 499.329749] Lustre: 24283:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 499.333301] Lustre: 24283:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 499.338016] Lustre: 24283:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 499.344210] Lustre: 24283:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 499.349273] Lustre: 24283:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 499.354195] Lustre: 24283:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 499.361096] Lustre: 24283:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 499.367379] Lustre: 24283:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 499.374480] Lustre: 24283:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 499.379309] Lustre: 24283:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 499.383862] Lustre: 24283:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 509.397452] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 509.403921] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 509.408234] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 509.415242] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 509.419863] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 509.424420] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 509.428447] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 509.432418] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 509.436570] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 509.440334] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 509.444529] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 509.448590] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 556.514321] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.518817] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.529617] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 560.294095] Lustre: server umount lustre-OST07c6 complete [ 565.472467] Lustre: server umount lustre-MDT0000 complete [ 574.002540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 575.477583] Key type lgssc unregistered [ 575.648104] LNet: 27798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.653427] LNetError: 27798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.666706] LNet: Removed LNI 192.168.204.146@tcp [ 576.056163] Key type .llcrypt unregistered [ 576.058289] Key type ._llcrypt unregistered [ 586.472301] Key type ._llcrypt registered [ 586.475539] Key type .llcrypt registered [ 586.538318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 587.093573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 587.101825] alg: No test for adler32 (adler32-zlib) [ 588.024615] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 588.172770] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 589.799175] Key type lgssc registered [ 590.410496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 594.210269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 594.220236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.487298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.685376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 602.073347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.266593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 604.106929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 606.827032] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 608.876605] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 612.416248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.516766] LustreError: 30623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.527931] LustreError: 30623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 612.634159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.777883] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 618.507843] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 620.011778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 631.778468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 631.781499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.786591] LustreError: Skipped 1 previous similar message [ 631.793358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.143372] Lustre: server umount lustre-OST0000 complete [ 638.437542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.439110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.443293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.451192] Lustre: Skipped 1 previous similar message [ 641.336636] Lustre: server umount lustre-MDT0000 complete [ 643.379838] LustreError: 29013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327464 with bad export cookie 17234306971897785992 [ 643.382888] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.388856] LustreError: 29013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 653.878299] Lustre: server umount lustre-MDT0001 complete [ 659.989945] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 661.655882] Key type lgssc unregistered [ 661.866659] LNet: 32409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.872557] LNetError: 32409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.885104] LNet: Removed LNI 192.168.204.146@tcp [ 662.380541] Key type .llcrypt unregistered [ 662.382186] Key type ._llcrypt unregistered [ 672.613458] Key type ._llcrypt registered [ 672.615438] Key type .llcrypt registered [ 672.684359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 673.158214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.234534] alg: No test for adler32 (adler32-zlib) [ 674.186458] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 674.308846] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 675.927286] Key type lgssc registered [ 676.535461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 681.063342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 681.077360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.461783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.259456] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 690.152991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.344563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 692.454349] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 695.774697] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.250105] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 702.401929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 702.593737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.011564] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 707.763242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 709.498738] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 720.356282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 720.356742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.361460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.368641] Lustre: Skipped 1 previous similar message [ 725.481519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 725.489313] Lustre: Skipped 2 previous similar messages [ 726.214765] Lustre: server umount lustre-OST0000 complete [ 730.593917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.595739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.607546] Lustre: Skipped 1 previous similar message [ 734.668845] Lustre: server umount lustre-MDT0000 complete [ 735.712472] LustreError: 33638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 735.724636] LustreError: 33638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 736.972825] LustreError: 33622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327558 with bad export cookie 236556431682688624 [ 736.974170] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.979165] LustreError: 33622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 737.187649] Lustre: server umount lustre-MDT0001 complete [ 743.267170] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 744.896792] Key type lgssc unregistered [ 745.074513] LNet: 36816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.080255] LNetError: 36816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.091060] LNet: Removed LNI 192.168.204.146@tcp [ 745.558418] Key type .llcrypt unregistered [ 745.560551] Key type ._llcrypt unregistered [ 759.088550] Key type ._llcrypt registered [ 759.090403] Key type .llcrypt registered [ 759.145278] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 759.702922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.716756] alg: No test for adler32 (adler32-zlib) [ 760.687473] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 760.845290] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 762.471166] Key type lgssc registered [ 763.188321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 767.755564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 767.768924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.044830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.417362] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 776.252596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.469380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 778.977589] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 782.422903] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 784.485963] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 788.357200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.603741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.663810] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 794.556266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.731528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 806.372306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.380969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 806.387701] Lustre: Skipped 2 previous similar messages [ 806.401857] Lustre: Skipped 2 previous similar messages [ 807.399652] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 809.594636] Lustre: server umount lustre-MDT0000 complete [ 810.394685] LustreError: 38189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 810.403354] LustreError: 38189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 811.490520] LustreError: 38188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.501149] LustreError: 38188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 811.588381] LustreError: 38173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327632 with bad export cookie 2728191100582789872 [ 811.592092] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 811.594724] LustreError: 38173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 815.587504] LustreError: 38189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.600525] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 816.610167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.821873] Lustre: server umount lustre-MDT0001 complete [ 836.690509] Lustre: server umount lustre-OST0000 complete [ 845.286358] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 846.809430] Key type lgssc unregistered [ 846.988507] LNet: 41693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.993564] LNetError: 41693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.008315] LNet: Removed LNI 192.168.204.146@tcp [ 847.432653] Key type .llcrypt unregistered [ 847.436197] Key type ._llcrypt unregistered [ 859.766170] Key type ._llcrypt registered [ 859.767790] Key type .llcrypt registered [ 859.827421] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 860.241562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.264457] alg: No test for adler32 (adler32-zlib) [ 861.208628] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 861.350053] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 862.959202] Key type lgssc registered [ 863.530162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.577244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.587913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.853961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.900771] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 875.475293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.662329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.691284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 880.320401] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 882.101619] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 885.614837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.813095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.691834] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 891.239496] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 893.996393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 909.280687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.286357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.292643] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 909.304486] Lustre: Skipped 1 previous similar message [ 910.306449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.313683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 910.314638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.548774] Lustre: server umount lustre-OST0000 complete [ 925.665455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.666510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.672461] Lustre: Skipped 1 previous similar message [ 925.679528] Lustre: Skipped 1 previous similar message [ 930.784416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.786887] Lustre: Skipped 2 previous similar messages [ 931.830923] Lustre: server umount lustre-MDT0000 complete [ 933.831394] LustreError: 43051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327755 with bad export cookie 8178516069647464347 [ 933.833849] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.838517] LustreError: 43051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 948.191203] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 948.290708] Lustre: server umount lustre-MDT0001 complete [ 952.826336] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 954.138890] Key type lgssc unregistered [ 954.305959] LNet: 46446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.311723] LNetError: 46446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.323517] LNet: Removed LNI 192.168.204.146@tcp [ 954.769975] Key type .llcrypt unregistered [ 954.771366] Key type ._llcrypt unregistered [ 966.203395] Key type ._llcrypt registered [ 966.205648] Key type .llcrypt registered [ 966.263976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 966.658358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.680443] alg: No test for adler32 (adler32-zlib) [ 967.607950] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 967.757072] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 969.367292] Key type lgssc registered [ 969.926114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.752142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.768840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.047823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.276645] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 981.817134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.018419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 983.849708] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 986.558893] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 988.344177] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 991.729931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.898527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.769118] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 997.468657] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1000.108600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1006.628879] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1008.708459] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1021.037546] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0111h30m40s [ 1023.229984] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1026.016835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.021061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.032177] Lustre: Skipped 2 previous similar messages [ 1026.043489] Lustre: Skipped 2 previous similar messages [ 1027.106990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.624414] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1029.637189] Lustre: Skipped 2 previous similar messages [ 1031.613289] LustreError: 50377:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000963d4f45) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1033.185222] LustreError: 48887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.227495] LustreError: 48887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1033.724142] Lustre: server umount lustre-MDT0000 complete [ 1035.758480] LustreError: 48886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.643392] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1039.336679] LustreError: 47818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.366733] LustreError: 47818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1041.993606] LustreError: 48886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.023459] LustreError: 47817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.038427] LustreError: 47817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1048.319608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.400647] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.888456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.042073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1049.909762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1054.909440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1059.955371] Lustre: DEBUG MARKER: Wait for df (36066) ... [ 1061.526906] Lustre: DEBUG MARKER: done [ 1065.445275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.461807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.465754] Lustre: Skipped 2 previous similar messages [ 1070.593505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.603153] Lustre: Skipped 1 previous similar message [ 1071.788275] Lustre: server umount lustre-OST0000 complete [ 1080.800674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1080.804695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.806606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.819874] Lustre: Skipped 2 previous similar messages [ 1083.324813] Lustre: server umount lustre-MDT0000 complete [ 1087.148911] LustreError: 47803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780327908 with bad export cookie 18292125052293970680 [ 1087.160042] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1087.173375] LustreError: 47803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1087.931782] Lustre: server umount lustre-MDT0001 complete [ 1101.073065] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1106.123822] Key type lgssc unregistered [ 1106.605688] LNet: 52371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.620522] LNetError: 52371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.684335] LNet: Removed LNI 192.168.204.146@tcp [ 1107.764375] Key type .llcrypt unregistered [ 1107.767403] Key type ._llcrypt unregistered [ 1154.099946] Key type ._llcrypt registered [ 1154.104901] Key type .llcrypt registered [ 1154.271144] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1156.836657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.881177] alg: No test for adler32 (adler32-zlib) [ 1158.162750] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1158.605326] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1160.575388] Key type lgssc registered [ 1161.907188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1170.930153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1170.959526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.478754] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1172.512289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1172.754868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.876563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1185.433261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.541269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1185.563643] Lustre: 54572:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1185.811322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.067617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1195.063973] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1198.515475] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1205.138105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.395391] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1205.571189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.629526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1215.623914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1216.536542] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.798506] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0111h34m03s [ 1225.386448] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1231.786248] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1234.294290] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1234.349955] LustreError: 54580:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1254.412351] Lustre: lustre-MDT0000: Client 1f35b4c1-4336-4f5d-9c81-0325febb1e38 (at 192.168.204.46@tcp) reconnecting [ 1274.376076] LustreError: 54580:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1276.716744] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1279.152123] Lustre: DEBUG MARKER: done [ 1286.874875] Lustre: server umount lustre-OST0000 complete [ 1292.778873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.786667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.797044] Lustre: Skipped 1 previous similar message [ 1292.810550] Lustre: Skipped 1 previous similar message [ 1293.862626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.949455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.955021] Lustre: Skipped 2 previous similar messages [ 1300.006696] Lustre: server umount lustre-MDT0000 complete [ 1303.012053] Lustre: 53891:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328108/real 1780328108] req@000000009bbee440 x1866809190960512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780328124 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1303.063149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.955721] LustreError: 54564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328125 with bad export cookie 10582897645926018835 [ 1303.977174] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.442683] Lustre: server umount lustre-MDT0001 complete [ 1316.514922] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1319.781700] Key type lgssc unregistered [ 1320.123439] LNet: 58548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.139399] LNetError: 58548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.195953] LNet: Removed LNI 192.168.204.146@tcp [ 1322.168739] Key type .llcrypt unregistered [ 1322.172565] Key type ._llcrypt unregistered [ 1369.712616] Key type ._llcrypt registered [ 1369.716181] Key type .llcrypt registered [ 1369.962233] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1370.950929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.059196] alg: No test for adler32 (adler32-zlib) [ 1372.415942] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1372.820409] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1374.535706] Key type lgssc registered [ 1376.038603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1380.009686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1380.024656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.262782] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1381.287266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1381.478795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.957602] Lustre: Failing over lustre-MDT0000 [ 1383.209283] Lustre: server umount lustre-MDT0000 complete [ 1394.788137] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:36:53 (1780328213) [ 1404.897428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.782262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.641364] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1423.153342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.274441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1423.306822] Lustre: 61330:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1423.610946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1429.666140] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1435.157898] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1439.350807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1448.734686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1449.065097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1449.537428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.283869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1457.256097] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1462.378885] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1468.250298] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1477.091086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.096776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.141353] Lustre: Skipped 1 previous similar message [ 1477.163437] Lustre: Skipped 2 previous similar messages [ 1482.405493] Lustre: server umount lustre-MDT0000 complete [ 1487.330651] LustreError: 62256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.367419] LustreError: 62256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1487.557777] LustreError: 61324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328308 with bad export cookie 5381754491260708526 [ 1487.562055] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.580455] LustreError: 61324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1488.267412] Lustre: server umount lustre-MDT0001 complete [ 1491.993239] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1496.286407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1504.113019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.807635] Lustre: 60252:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328313/real 1780328313] req@00000000abf3fded x1866809416394112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1507.862534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.879331] Lustre: Skipped 1 previous similar message [ 1511.976550] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4aafd5322ff52f98 [ 1511.996912] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 1512.388315] LustreError: 62959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.499335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.628497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1513.567263] Lustre: 60252:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328318/real 1780328318] req@00000000ea99a3eb x1866809416394240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1517.543354] LustreError: 62960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.322347] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1518.561070] Lustre: 60255:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328323/real 1780328323] req@000000006497bde9 x1866809416394496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1518.604245] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1523.696302] LustreError: 62959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.731463] LustreError: 62959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1524.707099] Lustre: 60254:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328329/real 1780328329] req@0000000020baad14 x1866809416394624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1526.885968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.310679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.257129] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1532.411571] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1536.275320] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1539.496841] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1546.617928] Lustre: server umount lustre-MDT0000 complete [ 1547.748513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.756174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.758316] LustreError: 64766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.772128] Lustre: Skipped 1 previous similar message [ 1549.607856] LustreError: 61324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328370 with bad export cookie 5381754491260710808 [ 1549.609398] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.620304] LustreError: 61324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1549.824488] Lustre: server umount lustre-MDT0001 complete [ 1555.269557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1563.526303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.247167] Lustre: 60255:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328374/real 1780328374] req@0000000009f861b6 x1866809416418944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1569.268145] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.279250] Lustre: Skipped 1 previous similar message [ 1574.682552] LustreError: 62959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1574.693307] LustreError: 62959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1574.762678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.799834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1577.754204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1580.448100] Lustre: 60255:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780328385/real 1780328385] req@0000000062df3a5e x1866809416419584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780328401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1580.474151] Lustre: 60255:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1584.559864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.010718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1588.123217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1591.802916] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.734156] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1600.480420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1600.487807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.503760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.733460] Lustre: server umount lustre-MDT0000 complete [ 1603.555853] LustreError: 67224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.573842] LustreError: 67224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1604.632256] LustreError: 61322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328425 with bad export cookie 5381754491260712187 [ 1604.638384] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.649921] LustreError: 61322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1604.912971] Lustre: server umount lustre-MDT0001 complete [ 1610.101322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1612.904483] Lustre: DEBUG MARKER: files compared the same [ 1624.715956] Lustre: server umount lustre-OST0000 complete [ 1640.401930] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1643.167511] Key type lgssc unregistered [ 1643.388618] LNet: 70254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.393000] LNetError: 70254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.408195] LNet: Removed LNI 192.168.204.146@tcp [ 1644.150610] Key type .llcrypt unregistered [ 1644.152458] Key type ._llcrypt unregistered [ 1669.477710] Key type ._llcrypt registered [ 1669.482532] Key type .llcrypt registered [ 1669.672937] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1670.958448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.019404] alg: No test for adler32 (adler32-zlib) [ 1672.147997] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1672.479336] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1674.297285] Key type lgssc registered [ 1675.979299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.067524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1685.105233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.783024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.456756] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 1696.226169] LustreError: 71655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.245315] LustreError: 71655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1700.440594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.814251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1705.540257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 1711.644502] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1715.776770] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1723.190601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1723.505235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.297926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 1731.701725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1734.660959] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1746.400298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.412414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.437167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.425482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.426617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.437505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.734687] Lustre: server umount lustre-OST0000 complete [ 1757.152458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1757.160609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.190854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.195363] Lustre: Skipped 1 previous similar message [ 1759.912051] Lustre: server umount lustre-MDT0000 complete [ 1762.784083] LustreError: 73402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.807523] LustreError: 73402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1763.922950] LustreError: 71635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328585 with bad export cookie 4845194835579686982 [ 1763.923477] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.936839] LustreError: 71635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1764.493338] Lustre: server umount lustre-MDT0001 complete [ 1774.952286] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1777.833510] Key type lgssc unregistered [ 1778.148971] LNet: 74833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.158399] LNetError: 74833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.185411] LNet: Removed LNI 192.168.204.146@tcp [ 1779.052701] Key type .llcrypt unregistered [ 1779.056135] Key type ._llcrypt unregistered [ 1801.603568] Key type ._llcrypt registered [ 1801.605436] Key type .llcrypt registered [ 1801.729668] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1802.726149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.779953] alg: No test for adler32 (adler32-zlib) [ 1803.901476] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1804.104229] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1805.831305] Key type lgssc registered [ 1806.991823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1813.107401] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1813.127872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.703365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1833.824179] LustreError: 76222:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 1860.511463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.637616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1871.837728] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1879.878864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.079803] Lustre: *** cfs_fail_loc=706, val=0*** [ 1880.475770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.594240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1884.772439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1885.670475] LustreError: 76247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.694453] LustreError: 76247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1890.788453] LustreError: 76248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.407970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.774151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1898.451749] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1903.858785] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.333589] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1909.218981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1909.233330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.246715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.269411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.269913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.284992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.162488] Lustre: server umount lustre-OST0000 complete [ 1919.467877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1919.480415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.494136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.500392] Lustre: Skipped 1 previous similar message [ 1921.505139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.516421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.548759] Lustre: Skipped 1 previous similar message [ 1925.115492] Lustre: server umount lustre-MDT0000 complete [ 1926.625160] LustreError: 77195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.656316] LustreError: 77195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1929.872815] LustreError: 76229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780328751 with bad export cookie 8821689957301344390 [ 1929.874226] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.885190] LustreError: 76229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.365812] Lustre: server umount lustre-MDT0001 complete [ 1942.268434] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1945.263532] Key type lgssc unregistered [ 1945.837077] LNet: 79481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.842552] LNetError: 79481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.874289] LNet: Removed LNI 192.168.204.146@tcp [ 1947.009548] Key type .llcrypt unregistered [ 1947.014566] Key type ._llcrypt unregistered [ 1970.777655] Key type ._llcrypt registered [ 1970.779702] Key type .llcrypt registered [ 1970.868405] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1971.815434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.837450] alg: No test for adler32 (adler32-zlib) [ 1973.035329] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1973.185077] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1974.807208] Key type lgssc registered [ 1975.862156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.748231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.766693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.497825] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1994.953907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.345596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.993706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2000.360417] LustreError: 81551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.374494] LustreError: 81551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2005.475813] LustreError: 81550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.491583] LustreError: 81550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2007.077129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.426039] LustreError: 81551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.441327] LustreError: 81551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2007.448182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.642296] LustreError: 81555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.661422] LustreError: 81555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2012.771611] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2017.762922] LustreError: 81554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.776646] LustreError: 81554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2019.144230] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.164453] Lustre: MGS: Client a7ff02e4-0e22-43f1-8629-1d464edb4b05 (at 0@lo) reconnecting [ 2019.190988] LustreError: 81565:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2019.345681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.418830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2022.768033] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2034.660033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.669953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.682527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2035.685421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2035.685577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2035.685723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.695661] Lustre: Skipped 1 previous similar message [ 2039.901635] Lustre: server umount lustre-OST0000 complete [ 2045.920352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.926353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.934565] Lustre: Skipped 1 previous similar message [ 2045.935538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2045.938219] Lustre: Skipped 1 previous similar message [ 2048.762362] Lustre: server umount lustre-MDT0000 complete [ 2051.045028] LustreError: 82284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.065652] LustreError: 82284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2051.920595] Lustre: server umount lustre-MDT0001 complete [ 2055.023857] Lustre: server umount MGS complete [ 2063.294752] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:48:03 (1780328883) [ 2083.400762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2089.394103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2112.817036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.551112] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2121.536669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.663772] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2121.694600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2121.746617] Lustre: lustre-MDT0001: new disk, initializing [ 2121.792790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.807084] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2121.814438] Lustre: 88617:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2121.820908] LustreError: 88617:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2121.833426] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2121.844496] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2122.334173] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2122.342971] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) Skipped 5393 previous similar messages [ 2122.348122] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2122.353214] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 5393 previous similar messages [ 2123.334027] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2123.341328] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) Skipped 10833 previous similar messages [ 2123.348057] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2123.354780] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 10849 previous similar messages [ 2124.026083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2125.334036] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2125.339294] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) Skipped 22407 previous similar messages [ 2125.348353] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2125.353594] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 22458 previous similar messages [ 2129.334798] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2129.340634] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) Skipped 46039 previous similar messages [ 2129.348751] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2129.353033] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 46020 previous similar messages [ 2130.553662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.778917] Lustre: lustre-OST0000: new disk, initializing [ 2130.787940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2130.794229] Lustre: 89365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2130.846303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.987023] Lustre: 89539:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d5d50fd7 x1866810045538304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780328971 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2135.150716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2137.334390] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2137.339988] LustreError: 88619:0:(fld_handler.c:275:fld_server_lookup()) Skipped 74212 previous similar messages [ 2137.348118] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2137.352849] LustreError: 88619:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 74201 previous similar messages [ 2140.053119] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.983293] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.991715] Lustre: MGS: Client 75fb9f7b-0a64-4ffc-85a9-dba03b251edf (at 0@lo) reconnecting [ 2144.026828] Lustre: 87900:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2144.042807] LustreError: 88603:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2144.052452] LustreError: 88603:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2144.082475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2144.136932] Lustre: lustre-MDT0000: new disk, initializing [ 2144.241457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.258073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2144.338753] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2145.312970] Lustre: 90322:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000cfb0ebea x1866810045544704/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780328982 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2146.530081] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2150.445373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2150.452803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2150.472242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2152.493502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2152.497576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2174.947360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2174.953770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.961264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.095067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.099491] Lustre: Skipped 3 previous similar messages [ 2186.207741] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2186.290391] Lustre: server umount lustre-OST0000 complete [ 2191.329137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.329274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.336048] Lustre: Skipped 1 previous similar message [ 2191.340123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.344678] Lustre: Skipped 3 previous similar messages [ 2194.383876] Lustre: server umount lustre-MDT0000 complete [ 2202.547402] Lustre: server umount lustre-MDT0001 complete [ 2204.278065] Lustre: server umount MGS complete [ 2209.273776] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:50:29 (1780329029) [ 2216.982894] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2218.286138] Key type lgssc unregistered [ 2218.436063] LNet: 92787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.439636] LNetError: 92787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.452497] LNet: Removed LNI 192.168.204.146@tcp [ 2218.810119] Key type .llcrypt unregistered [ 2218.811833] Key type ._llcrypt unregistered [ 2227.200386] Key type ._llcrypt registered [ 2227.201789] Key type .llcrypt registered [ 2227.254341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2227.678623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.729381] alg: No test for adler32 (adler32-zlib) [ 2228.617728] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2228.734475] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2230.327188] Key type lgssc registered [ 2230.889495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2234.749612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2234.762730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.994229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.774211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2241.727421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.907927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2243.760488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2246.190235] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2247.930988] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2251.117835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2251.273977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.888229] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2256.664627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2261.544152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2266.593414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.594978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.598079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.603867] Lustre: Skipped 1 previous similar message [ 2270.677459] Lustre: server umount lustre-OST0000 complete [ 2272.735874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.739297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.746080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.748759] Lustre: Skipped 1 previous similar message [ 2276.832083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.832644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.471379] Lustre: server umount lustre-MDT0000 complete [ 2280.357827] LustreError: 93950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329101 with bad export cookie 14311551803697107075 [ 2280.359424] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.364628] LustreError: 93950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2280.527495] Lustre: server umount lustre-MDT0001 complete [ 2285.439317] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2286.706405] Key type lgssc unregistered [ 2286.854989] LNet: 97144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.858991] LNetError: 97144:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.873835] LNet: Removed LNI 192.168.204.146@tcp [ 2287.237483] Key type .llcrypt unregistered [ 2287.239415] Key type ._llcrypt unregistered [ 2295.319524] Key type ._llcrypt registered [ 2295.321353] Key type .llcrypt registered [ 2295.430969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2295.991342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2296.009982] alg: No test for adler32 (adler32-zlib) [ 2296.960233] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2297.126356] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2298.791983] Key type lgssc registered [ 2299.356803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.285175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.288663] LustreError: 98187:0:(tgt_mount.c:2525:server_fill_super()) cfs_race id 716 sleeping [ 2302.291168] LustreError: 98189:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 waking [ 2302.296892] LustreError: 98187:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2302.300546] LustreError: 98187:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2302.307756] LustreError: 98187:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2302.309991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.598570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.145824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.314745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.306551] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2311.769669] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lsmod [ 2313.230665] LustreError: 99360:0:(tgt_mount.c:2525:server_fill_super()) cfs_race id 716 sleeping [ 2313.231177] LustreError: 99361:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 waking [ 2313.241785] LustreError: 99360:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2313.247965] LustreError: 99360:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2313.255331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2313.255821] LustreError: 99360:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2313.500851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.537667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2316.961577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.964901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.971871] Lustre: Skipped 2 previous similar messages [ 2316.980942] Lustre: Skipped 2 previous similar messages [ 2321.129099] Lustre: server umount lustre-MDT0000 complete [ 2322.404843] LustreError: 98226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.419133] LustreError: 98226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2323.037486] LustreError: 98207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329144 with bad export cookie 17719193610291788912 [ 2323.040637] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.044517] LustreError: 98207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2323.103317] LustreError: 98658:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002485506d x1866810385255296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2323.191513] Lustre: server umount lustre-MDT0001 complete [ 2335.791863] Lustre: server umount lustre-OST0000 complete [ 2339.405929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.630672] LustreError: 100427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.667066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.282199] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2344.929193] LustreError: 100428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.066357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.218525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.800460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2349.049512] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2350.642101] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2353.481440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.652054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.030704] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2358.341923] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2363.885931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 2371.040575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2371.043944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.050948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.112898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.114077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.116911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.124871] Lustre: server umount lustre-OST0000 complete [ 2379.234981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.245648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.249342] Lustre: Skipped 2 previous similar messages [ 2384.353073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.356094] Lustre: Skipped 1 previous similar message [ 2385.078658] Lustre: server umount lustre-MDT0000 complete [ 2386.700299] LustreError: 100407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329207 with bad export cookie 17719193610291789829 [ 2386.703508] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.704920] LustreError: 100407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2386.858985] Lustre: server umount lustre-MDT0001 complete [ 2391.084603] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2392.269851] Key type lgssc unregistered [ 2392.410465] LNet: 103622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.413831] LNetError: 103622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.423338] LNet: Removed LNI 192.168.204.146@tcp [ 2392.770753] Key type .llcrypt unregistered [ 2392.772579] Key type ._llcrypt unregistered [ 2402.861672] Key type ._llcrypt registered [ 2402.863881] Key type .llcrypt registered [ 2402.910450] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2403.263196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.271664] alg: No test for adler32 (adler32-zlib) [ 2404.141410] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2404.239674] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2405.831183] Key type lgssc registered [ 2406.254535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2409.458512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2409.467865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.678393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.257416] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2415.789617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.971891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2417.583464] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2419.620056] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2420.977150] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2423.500910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.633542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.664869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2427.602812] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2434.918964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2441.049246] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2441.855592] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2445.281721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.282810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.285936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.291376] Lustre: Skipped 1 previous similar message [ 2445.295988] Lustre: Skipped 1 previous similar message [ 2449.106373] Lustre: server umount lustre-OST0000 complete [ 2451.936383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.939416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.944958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.520740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.521111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.596395] Lustre: server umount lustre-MDT0000 complete [ 2457.910512] LustreError: 104980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329279 with bad export cookie 656936675154915950 [ 2457.913042] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.915502] LustreError: 104980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2458.037604] Lustre: server umount lustre-MDT0001 complete [ 2461.708847] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2462.825871] Key type lgssc unregistered [ 2462.961719] LNet: 108304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.966953] LNetError: 108304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.977417] LNet: Removed LNI 192.168.204.146@tcp [ 2463.291960] Key type .llcrypt unregistered [ 2463.293285] Key type ._llcrypt unregistered [ 2471.047335] Key type ._llcrypt registered [ 2471.048928] Key type .llcrypt registered [ 2471.097849] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2471.468479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.516942] alg: No test for adler32 (adler32-zlib) [ 2472.439854] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2472.550834] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2474.143159] Key type lgssc registered [ 2474.612181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.976460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.984724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.193385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.873131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2484.481788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.688909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2486.370471] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2488.814356] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.392478] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2493.357510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.527226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.046563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2498.392268] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2505.828260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2515.425086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2515.427866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.433815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.448229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.450229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.450887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.457139] Lustre: Skipped 1 previous similar message [ 2517.658992] Lustre: server umount lustre-OST0000 complete [ 2520.544751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.548526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.553723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.688330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.690971] Lustre: Skipped 3 previous similar messages [ 2531.809579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.812342] Lustre: Skipped 1 previous similar message [ 2533.344404] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2533.423603] Lustre: server umount lustre-MDT0000 complete [ 2534.868604] LustreError: 109468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329356 with bad export cookie 15168005669306120537 [ 2534.873450] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.875289] LustreError: 109468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2534.994743] Lustre: server umount lustre-MDT0001 complete [ 2539.053257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2540.262804] Key type lgssc unregistered [ 2540.400506] LNet: 112686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.403677] LNetError: 112686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.411683] LNet: Removed LNI 192.168.204.146@tcp [ 2540.755558] Key type .llcrypt unregistered [ 2540.757166] Key type ._llcrypt unregistered [ 2551.147297] Key type ._llcrypt registered [ 2551.148590] Key type .llcrypt registered [ 2551.200073] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2551.640397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.701211] alg: No test for adler32 (adler32-zlib) [ 2552.593299] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2552.695521] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2554.295383] Key type lgssc registered [ 2554.810648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.441106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.452991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.695734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.293522] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2564.990960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.183306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2566.907712] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2569.203570] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2570.711168] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2573.522536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.663117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.711266] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2577.593276] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.914945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 2590.817853] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2592.161123] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2592.163485] Lustre: Skipped 2 previous similar messages [ 2593.800718] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2593.804080] Lustre: Skipped 2 previous similar messages [ 2600.453417] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2600.471993] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2604.209180] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2604.214443] Lustre: Skipped 2 previous similar messages [ 2606.143217] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp [ 2611.488529] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.46/24@tcp in log lustre-MDT0000 [ 2611.492653] Lustre: Skipped 2 previous similar messages [ 2613.886284] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 2613.888836] Lustre: Skipped 1 previous similar message [ 2619.873586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.873915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.875746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.884443] Lustre: Skipped 1 previous similar message [ 2622.940171] Lustre: server umount lustre-OST0000 complete [ 2624.993195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.993916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.000610] Lustre: Skipped 1 previous similar message [ 2625.003488] Lustre: Skipped 1 previous similar message [ 2630.112907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.116160] Lustre: Skipped 2 previous similar messages [ 2630.567458] Lustre: server umount lustre-MDT0000 complete [ 2632.149527] LustreError: 117214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329453 with bad export cookie 3904652520432999173 [ 2632.152492] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.155442] LustreError: 117214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2632.290158] Lustre: server umount lustre-MDT0001 complete [ 2636.587446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2637.856626] Key type lgssc unregistered [ 2638.007550] LNet: 118705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.011710] LNetError: 118705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.023510] LNet: Removed LNI 192.168.204.146@tcp [ 2638.420808] Key type .llcrypt unregistered [ 2638.422313] Key type ._llcrypt unregistered [ 2649.989910] Key type ._llcrypt registered [ 2649.991459] Key type .llcrypt registered [ 2650.034479] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2650.410423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.446492] alg: No test for adler32 (adler32-zlib) [ 2651.314248] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2651.410503] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2652.999251] Key type lgssc registered [ 2653.425728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.421097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2656.431912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.650661] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2657.659872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.663308] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.46/24@tcp [ 2659.238699] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2662.486567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.616520] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2662.626331] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 2662.642474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2664.169310] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2666.393864] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2667.792908] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2670.296667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.466310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.532877] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2674.513442] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2682.790882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 2693.088785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2693.091661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.092064] LustreError: Skipped 1 previous similar message [ 2693.096687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2693.099930] Lustre: Skipped 1 previous similar message [ 2694.423296] Lustre: server umount lustre-OST0000 complete [ 2698.208679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.209472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.212987] Lustre: Skipped 2 previous similar messages [ 2701.979323] Lustre: server umount lustre-MDT0000 complete [ 2703.330044] LustreError: 120225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.337497] LustreError: 120225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2703.463087] LustreError: 121846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329524 with bad export cookie 10790955803637091347 [ 2703.464778] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2703.468290] LustreError: 121846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2703.598072] Lustre: server umount lustre-MDT0001 complete [ 2707.888560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2709.180332] Key type lgssc unregistered [ 2709.334329] LNet: 123476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.338262] LNetError: 123476:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2709.348823] LNet: Removed LNI 192.168.204.146@tcp [ 2709.736672] Key type .llcrypt unregistered [ 2709.738471] Key type ._llcrypt unregistered [ 2719.322763] Key type ._llcrypt registered [ 2719.324262] Key type .llcrypt registered [ 2719.366969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2725.523033] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2725.812100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2725.826078] alg: No test for adler32 (adler32-zlib) [ 2726.683411] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2726.776884] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2728.359200] Key type lgssc registered [ 2728.784206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.711200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.722676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.831580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2742.845372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2742.884699] Lustre: lustre-MDT0000: new disk, initializing [ 2742.915795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.925033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2744.413351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2749.053137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.098267] Lustre: 126815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2749.124749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2749.126706] Lustre: Skipped 1 previous similar message [ 2749.163996] Lustre: lustre-MDT0001: new disk, initializing [ 2749.218245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.229628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2749.232623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2750.653188] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2754.068513] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2755.527239] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2758.140425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.265386] Lustre: lustre-OST0000: new disk, initializing [ 2758.267510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2758.271731] Lustre: 128560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2758.311951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.820295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2758.824854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2758.871913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2760.461166] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2763.884157] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2765.324583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2765.412949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2766.709241] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2766.793530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2769.380109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.380520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2769.381841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2769.384902] Lustre: Skipped 1 previous similar message [ 2769.393210] LustreError: Skipped 1 previous similar message [ 2773.599557] Lustre: server umount lustre-OST0000 complete [ 2775.008132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.012092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.018644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.021363] Lustre: Skipped 1 previous similar message [ 2779.616813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.617416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.736305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.739055] Lustre: Skipped 2 previous similar messages [ 2788.832584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.835795] Lustre: Skipped 1 previous similar message [ 2789.343140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2789.422903] Lustre: server umount lustre-MDT0000 complete [ 2790.856797] LustreError: 127594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329612 with bad export cookie 5405534872897268334 [ 2790.860264] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.865721] LustreError: 127594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2791.005611] Lustre: server umount lustre-MDT0001 complete [ 2794.238953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.439179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.844546] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2798.860827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.980440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.324895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2802.161601] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2803.348438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2805.614447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.636990] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2809.456648] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2810.969657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2819.214607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2820.671623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2820.763244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2824.365146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.439759] Lustre: lustre-OST0001: new disk, initializing [ 2824.441904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2824.445322] Lustre: 133683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2824.475575] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.479132] Lustre: Skipped 1 previous similar message [ 2826.536696] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2829.935769] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2833.403422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2833.406202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2833.428401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2834.074726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2835.319195] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2835.396588] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2836.503348] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ed8410c000.ost_server_uuid 50 [ 2836.976491] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88ed8410c000.ost_server_uuid in FULL state after 0 sec [ 2843.619035] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2843.619804] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.622273] LustreError: Skipped 1 previous similar message [ 2843.626413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2843.628991] Lustre: Skipped 1 previous similar message [ 2844.921165] Lustre: server umount lustre-OST0001 complete [ 2848.738227] LustreError: 132224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.738284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.743778] LustreError: 132224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2848.748956] Lustre: Skipped 2 previous similar messages [ 2852.257964] Lustre: server umount lustre-OST0000 complete [ 2855.392930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.978753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.995019] Lustre: Skipped 1 previous similar message [ 2861.042960] Lustre: server umount lustre-MDT0000 complete [ 2863.850888] LustreError: 130611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780329685 with bad export cookie 5405534872897269244 [ 2863.858904] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.866373] LustreError: 130611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2864.162568] Lustre: server umount lustre-MDT0001 complete [ 2880.652601] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2884.731580] Key type lgssc unregistered [ 2885.793924] LNet: 136558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2885.812150] LNetError: 136558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.824917] LNet: Removed LNI 192.168.204.146@tcp [ 2890.033362] Key type .llcrypt unregistered [ 2890.035315] Key type ._llcrypt unregistered [ 2976.416391] Key type ._llcrypt registered [ 2976.422433] Key type .llcrypt registered [ 2976.590887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2994.159918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2994.920241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2994.931407] alg: No test for adler32 (adler32-zlib) [ 2996.084803] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 2996.318315] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2998.057149] Key type lgssc registered [ 3000.193301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3037.120603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3037.149965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.611214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3038.659424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3038.875193] Lustre: lustre-MDT0000: new disk, initializing [ 3039.052421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.117581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3044.425169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3055.667767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.839354] Lustre: 140689:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3055.914506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3055.918421] Lustre: Skipped 1 previous similar message [ 3056.038810] Lustre: lustre-MDT0001: new disk, initializing [ 3056.146807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3056.188298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3056.203410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3060.981105] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3068.614447] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.387736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.560601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.879915] Lustre: lustre-OST0000: new disk, initializing [ 3079.883733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3079.902172] Lustre: 142434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3080.000886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.896148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3085.911259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3086.031763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3086.072818] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3094.875909] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3101.126539] Lustre: Failing over lustre-OST0000 [ 3101.419585] Lustre: server umount lustre-OST0000 complete [ 3117.026842] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329922/real 1780329922] req@00000000c2253326 x1866811118226944/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1780329938 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3117.031957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.056549] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3120.607864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.969781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.990778] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3121.955427] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329927/real 1780329927] req@00000000c40b1a7a x1866811118227968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780329943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3122.012113] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3127.077657] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3127.917857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3128.087096] Lustre: 138912:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329933/real 1780329933] req@0000000069959f9e x1866811118229248/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780329949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3128.107875] Lustre: 138912:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3128.383353] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3128.384344] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3131.321219] Lustre: Failing over lustre-MDT0000 [ 3131.620798] Lustre: server umount lustre-MDT0000 complete [ 3132.391094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.414450] Lustre: Skipped 3 previous similar messages [ 3137.504148] LustreError: 140700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.526245] LustreError: 140700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3138.529684] LustreError: 143572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.632498] LustreError: 140694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.650862] LustreError: 140694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3147.749941] LustreError: 140700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.759348] LustreError: 140700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3148.577295] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780329953/real 1780329953] req@000000006cc627a7 x1866811118239232/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780329969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3148.613539] Lustre: 138914:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3148.623738] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.357248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.883170] LustreError: 142035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.912708] LustreError: 142035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3158.373442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.472643] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3160.035045] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3163.633465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3163.644736] Lustre: Skipped 1 previous similar message [ 3163.686785] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3163.751524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3163.779224] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3173.862139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.869360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3173.890600] Lustre: Skipped 1 previous similar message [ 3173.907283] Lustre: Skipped 1 previous similar message [ 3178.936663] Lustre: server umount lustre-OST0000 complete [ 3184.096479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.099703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.106347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.111611] Lustre: Skipped 1 previous similar message [ 3188.610948] Lustre: server umount lustre-MDT0000 complete [ 3189.221145] LustreError: 141748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.242844] LustreError: 141748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3192.625969] LustreError: 140678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330013 with bad export cookie 13214087171083465079 [ 3192.633246] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.636705] LustreError: 140678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3193.131159] Lustre: server umount lustre-MDT0001 complete [ 3209.706543] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3214.178135] Key type lgssc unregistered [ 3214.675704] LNet: 145927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3214.682593] LNetError: 145927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3214.694264] LNet: Removed LNI 192.168.204.146@tcp [ 3215.567258] Key type .llcrypt unregistered [ 3215.571075] Key type ._llcrypt unregistered [ 3256.415517] Key type ._llcrypt registered [ 3256.420916] Key type .llcrypt registered [ 3256.616974] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3258.704077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3258.868333] alg: No test for adler32 (adler32-zlib) [ 3260.341599] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 3260.546581] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3262.568488] Key type lgssc registered [ 3264.790215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3280.023342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3280.059449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.900449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.497246] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3292.130936] LustreError: 147434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.138820] LustreError: 147434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3297.272510] LustreError: 147433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.990294] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.379101] LustreError: 147434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.667446] LustreError: 147462:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3304.688098] LustreError: 147462:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3304.698396] LustreError: 147462:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3305.193134] Lustre: server umount lustre-MDT0000 complete [ 3321.062364] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3331.153940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3359.607539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.777618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3359.811540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3359.860528] Lustre: lustre-MDT0000: new disk, initializing [ 3359.947546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.972623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3363.198855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3372.465590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.579840] Lustre: 150622:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3372.626955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3372.630235] Lustre: Skipped 1 previous similar message [ 3372.741466] Lustre: lustre-MDT0001: new disk, initializing [ 3372.831027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.865491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3372.876834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3376.156391] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3382.233411] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3385.264546] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3390.336256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.558334] Lustre: lustre-OST0000: new disk, initializing [ 3390.561725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3390.567203] Lustre: 152365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3390.652877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.900337] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3396.656449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3396.669181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3396.762285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3400.650255] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.585275] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3405.747731] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:10:25 (1780330225) [ 3566.048274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.051420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.057659] Lustre: Skipped 2 previous similar messages [ 3566.068116] Lustre: Skipped 2 previous similar messages [ 3571.168370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.174215] Lustre: Skipped 2 previous similar messages [ 3571.332928] Lustre: server umount lustre-MDT0000 complete [ 3573.284351] LustreError: 150614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330394 with bad export cookie 17727542631461097652 [ 3573.284871] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3573.292110] LustreError: 150614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3576.287814] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.288462] LustreError: 154000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.300342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3578.335761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3579.448528] Lustre: server umount lustre-MDT0001 complete [ 3587.438955] Lustre: server umount lustre-OST0000 complete [ 3592.902303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3597.770360] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3616.105145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.253646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3616.279653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3616.326444] Lustre: lustre-MDT0000: new disk, initializing [ 3616.373109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.384175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3618.230237] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3623.148149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.194343] Lustre: 156857:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.221713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.225058] Lustre: Skipped 1 previous similar message [ 3623.264485] Lustre: lustre-MDT0001: new disk, initializing [ 3623.314468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3623.319772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3624.944083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3628.235718] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3629.541148] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3632.201785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3632.301706] Lustre: lustre-OST0000: new disk, initializing [ 3632.303490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3632.305899] Lustre: 158601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3632.336621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.339648] Lustre: Skipped 1 previous similar message [ 3633.082476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3633.086669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3633.138093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3634.491213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3637.978346] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3639.479684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.568530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3641.072867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3641.165276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3643.360852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3643.362647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.364094] LustreError: Skipped 1 previous similar message [ 3643.371221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3643.374546] Lustre: Skipped 1 previous similar message [ 3643.996554] Lustre: server umount lustre-OST0000 complete [ 3648.480661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.484834] Lustre: Skipped 2 previous similar messages [ 3651.478387] Lustre: server umount lustre-MDT0000 complete [ 3652.756226] LustreError: 159992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330473 with bad export cookie 17727542631461256524 [ 3652.759444] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.761034] LustreError: 159992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3652.877342] Lustre: server umount lustre-MDT0001 complete [ 3655.742506] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:14:36 (1780330476) [ 3661.456158] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3664.708638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3678.301131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.415462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3678.431274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3678.475617] Lustre: lustre-MDT0000: new disk, initializing [ 3678.516520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.524598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3680.115585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3684.530301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.570780] Lustre: 163168:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.663845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3685.950628] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3689.017685] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3690.276705] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3692.417509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3692.520361] Lustre: 164923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3694.286330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3694.652260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3694.697176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3697.413498] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3699.681667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.681919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.682650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.682655] Lustre: Skipped 2 previous similar messages [ 3699.696657] Lustre: Skipped 1 previous similar message [ 3705.493176] Lustre: server umount lustre-OST0000 complete [ 3709.920655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.926763] Lustre: Skipped 1 previous similar message [ 3720.159954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.162136] Lustre: Skipped 8 previous similar messages [ 3721.183115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3721.244628] Lustre: server umount lustre-MDT0000 complete [ 3722.582320] LustreError: 163158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330543 with bad export cookie 17727542631461257434 [ 3722.584674] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.586884] LustreError: 163158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3722.720223] Lustre: server umount lustre-MDT0001 complete [ 3725.797376] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:15:46 (1780330546) [ 3732.850956] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3740.343528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3767.961917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.180776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3768.204578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3768.208875] Lustre: Skipped 3 previous similar messages [ 3768.275928] Lustre: lustre-MDT0000: new disk, initializing [ 3768.278458] Lustre: Skipped 2 previous similar messages [ 3768.343444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.347521] Lustre: Skipped 2 previous similar messages [ 3768.361441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3768.366074] Lustre: Skipped 2 previous similar messages [ 3771.656869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3780.518844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.630726] Lustre: 169088:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3780.933279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3784.100604] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3796.978791] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.884234] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3817.670966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3818.336751] Lustre: 170841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3824.776061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3824.953271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3827.067584] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3838.878578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3855.862843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.865310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.880064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.887452] LustreError: Skipped 1 previous similar message [ 3855.941635] Lustre: Skipped 1 previous similar message [ 3859.874216] Lustre: server umount lustre-OST0000 complete [ 3872.433407] Lustre: server umount lustre-MDT0000 complete [ 3876.322583] LustreError: 169097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.359631] LustreError: 169097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3878.408220] LustreError: 169078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330699 with bad export cookie 17727542631461258547 [ 3878.413698] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.424273] LustreError: 169078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3878.964609] Lustre: server umount lustre-MDT0001 complete [ 3893.150210] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3897.010449] Key type lgssc unregistered [ 3897.358063] LNet: 172570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.372283] LNetError: 172570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3897.395361] LNet: Removed LNI 192.168.204.146@tcp [ 3898.608762] Key type .llcrypt unregistered [ 3898.614906] Key type ._llcrypt unregistered [ 3924.073269] Key type ._llcrypt registered [ 3924.075142] Key type .llcrypt registered [ 3924.241939] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3925.592029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3925.636251] alg: No test for adler32 (adler32-zlib) [ 3926.731911] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 3927.019522] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3928.703215] Key type lgssc registered [ 3929.901996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3938.325872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3938.372660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.922452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.632697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3950.055545] LustreError: 173972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.075327] LustreError: 173972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3954.068101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.576336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3959.577348] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3965.717290] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3969.969531] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3977.193136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.614125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.567773] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3984.847731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3990.280768] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.472421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.472957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.485127] LustreError: Skipped 1 previous similar message [ 4010.494774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.511889] Lustre: Skipped 1 previous similar message [ 4010.550137] Lustre: Skipped 1 previous similar message [ 4015.591621] LustreError: 175571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.722582] Lustre: server umount lustre-OST0000 complete [ 4020.710752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.721082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.731044] Lustre: Skipped 1 previous similar message [ 4025.818209] Lustre: server umount lustre-MDT0000 complete [ 4025.824525] LustreError: 173973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.850232] LustreError: 173973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4030.034758] LustreError: 173954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330851 with bad export cookie 12760324678407408816 [ 4030.044586] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.049922] LustreError: 173954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4030.353719] Lustre: server umount lustre-MDT0001 complete [ 4040.872441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4043.933753] Key type lgssc unregistered [ 4044.238796] LNet: 177176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4044.254294] LNetError: 177176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4044.271193] LNet: Removed LNI 192.168.204.146@tcp [ 4045.046500] Key type .llcrypt unregistered [ 4045.054117] Key type ._llcrypt unregistered [ 4074.140414] Key type ._llcrypt registered [ 4074.143692] Key type .llcrypt registered [ 4074.250528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4075.823421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4075.875040] alg: No test for adler32 (adler32-zlib) [ 4076.972472] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 4077.211030] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4078.959172] Key type lgssc registered [ 4080.229218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4088.734667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4088.779519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.351721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.314638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4100.579422] LustreError: 178575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.593507] LustreError: 178575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4104.831382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.404324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4110.862348] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4117.146153] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4121.514636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4129.227031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.747050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.846185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4137.517766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4143.717944] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4150.249029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4150.265232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4150.286560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4150.802381] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 4151.264375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.270242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.336721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4153.344978] Lustre: Skipped 1 previous similar message [ 4155.880271] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 4155.893259] Lustre: Skipped 1 previous similar message [ 4160.987098] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 4160.994778] Lustre: Skipped 2 previous similar messages [ 4164.063992] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4164.339114] Lustre: server umount lustre-OST0000 complete [ 4170.814263] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88ed86464800.ost_server_uuid 50 [ 4172.682855] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4189.160722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.173538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.184272] Lustre: Skipped 3 previous similar messages [ 4193.595517] Lustre: server umount lustre-MDT0000 complete [ 4198.145045] LustreError: 178556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331019 with bad export cookie 1442656333289390789 [ 4198.159555] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.173804] LustreError: 178556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4198.708806] Lustre: server umount lustre-MDT0001 complete [ 4207.991826] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:23:47 (1780331027) [ 4215.279422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.744137] LustreError: 182056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.800208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.911285] LustreError: 182057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.917572] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4225.000442] LustreError: 182056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.027253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.118625] LustreError: 182057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.483489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4235.356633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4241.615837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4245.999295] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4252.967048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.353817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.696854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4260.548336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4265.828798] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4274.247281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.575630] Lustre: lustre-OST0001: new disk, initializing [ 4274.584544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4274.597247] Lustre: 184560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4274.722607] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4281.777378] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4284.055938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4284.069267] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4284.253311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4290.794507] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4294.627648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.650524] Lustre: Skipped 1 previous similar message [ 4294.654594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.658372] Lustre: Skipped 2 previous similar messages [ 4300.296914] Lustre: server umount lustre-OST0000 complete [ 4301.813459] LustreError: 185120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.649599] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4307.901348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4313.170448] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4313.426857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4320.227479] LustreError: 184031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.248488] LustreError: 184031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4330.472566] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4330.475088] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.496300] LustreError: Skipped 1 previous similar message [ 4330.499117] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4330.509087] Lustre: Skipped 2 previous similar messages [ 4330.511410] Lustre: Skipped 3 previous similar messages [ 4334.141435] Lustre: server umount lustre-OST0001 complete [ 4340.707957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.726363] Lustre: Skipped 1 previous similar message [ 4345.228648] Lustre: server umount lustre-MDT0000 complete [ 4350.216496] LustreError: 182036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331171 with bad export cookie 1442656333289391860 [ 4350.220036] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.232158] LustreError: 182036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4350.788261] Lustre: server umount lustre-MDT0001 complete [ 4376.894715] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:26:36 (1780331196) [ 4384.353035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.723428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4384.768815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4385.043449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.651367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4399.056370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.198690] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4399.262558] Lustre: 187857:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4404.070533] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4409.265523] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4413.355034] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4421.365912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.660113] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4428.943491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4429.138765] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4434.845878] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4442.249062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.341404] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4449.126826] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4454.973284] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4459.499902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4459.509186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.524628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4459.530372] Lustre: Skipped 3 previous similar messages [ 4472.287362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.427221] Lustre: server umount lustre-OST0000 complete [ 4475.897669] LustreError: 189482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.910887] LustreError: 189482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4490.726213] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4490.746077] LustreError: Skipped 1 previous similar message [ 4496.199369] Lustre: server umount lustre-OST0001 complete [ 4501.473685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.496182] Lustre: Skipped 4 previous similar messages [ 4506.937634] Lustre: server umount lustre-MDT0000 complete [ 4510.778695] LustreError: 189502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331331 with bad export cookie 1442656333289393267 [ 4510.793762] LustreError: 189502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4510.798135] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.131554] Lustre: server umount lustre-MDT0001 complete [ 4538.717262] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:29:17 (1780331357) [ 4555.517569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4564.688498] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4601.968987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.224791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4602.254595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4602.449936] Lustre: lustre-MDT0000: new disk, initializing [ 4602.635845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.640988] Lustre: Skipped 3 previous similar messages [ 4602.656323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4608.421531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4620.939157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.098180] Lustre: 195469:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4621.192946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4621.200112] Lustre: Skipped 1 previous similar message [ 4621.354414] Lustre: lustre-MDT0001: new disk, initializing [ 4621.589222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4621.619181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4626.269840] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4633.914859] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4637.812636] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4645.343231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.611912] Lustre: lustre-OST0000: new disk, initializing [ 4645.616438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4645.624685] Lustre: 197212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.119550] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4655.718444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4655.732969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4655.907679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4661.710721] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4666.754671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4667.099296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4671.700302] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4671.932473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4676.578986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.579458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4676.592853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4676.599089] LustreError: Skipped 1 previous similar message [ 4676.612373] Lustre: Skipped 17 previous similar messages [ 4679.764912] Lustre: server umount lustre-OST0000 complete [ 4689.789958] Lustre: server umount lustre-MDT0000 complete [ 4691.938780] LustreError: 195479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.959869] LustreError: 195479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4693.440372] LustreError: 197222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331514 with bad export cookie 1442656333289394835 [ 4693.453154] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.459606] LustreError: 197222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4693.905208] Lustre: server umount lustre-MDT0001 complete [ 4702.049458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.127771] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4714.925729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.977852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4725.874936] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4729.595422] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4736.422451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4736.798300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.806885] Lustre: Skipped 4 previous similar messages [ 4743.358672] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4748.928535] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4753.715746] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4753.950502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4759.241034] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4759.502272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4762.619988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4762.635218] LustreError: Skipped 1 previous similar message [ 4768.333496] Lustre: server umount lustre-OST0000 complete [ 4775.027338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4779.211906] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.319509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.180716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4818.800519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.823429] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.824630] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4818.859224] Lustre: Skipped 1 previous similar message [ 4818.903484] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4825.709480] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 4829.156601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.183056] Lustre: Skipped 5 previous similar messages [ 4829.189951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4835.402778] Lustre: server umount lustre-OST0000 complete [ 4853.215268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4853.418756] Lustre: server umount lustre-MDT0000 complete [ 4854.766527] LustreError: 199284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.826738] LustreError: 199284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4857.985367] LustreError: 199266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331679 with bad export cookie 1442656333289395759 [ 4857.998708] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.001179] LustreError: 199266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4858.691922] Lustre: server umount lustre-MDT0001 complete [ 4869.494860] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:34:48 (1780331688) [ 4875.710182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.136193] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4889.623443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.936491] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4900.355249] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4904.193724] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4910.895688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4917.181852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4922.350499] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4926.607304] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4926.783640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4930.368484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4930.530556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4936.274913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4936.443389] Lustre: lustre-OST0001: new disk, initializing [ 4936.446607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4936.453270] Lustre: 207729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4942.664573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4946.535183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4946.550622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4946.705829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4950.504756] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4955.445903] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4955.683851] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4960.607769] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4960.797387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4967.399765] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4967.405849] Lustre: Skipped 17 previous similar messages [ 4968.690631] Lustre: server umount lustre-OST0001 complete [ 4974.631334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4974.824353] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4979.158418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4979.435493] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5007.399709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5007.599207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.602509] Lustre: Skipped 5 previous similar messages [ 5012.582475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5012.973732] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5012.992741] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5012.993548] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5013.008631] Lustre: Skipped 1 previous similar message [ 5018.024377] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5019.750462] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5023.201483] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5027.715241] Lustre: server umount lustre-OST0001 complete [ 5049.758952] Lustre: server umount lustre-OST0000 complete [ 5059.666892] Lustre: server umount lustre-MDT0000 complete [ 5063.015487] LustreError: 204710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331884 with bad export cookie 1442656333289396725 [ 5063.022850] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5063.024152] LustreError: 204710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5069.446599] Lustre: server umount lustre-MDT0001 complete [ 5092.551549] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:38:32 (1780331912) [ 5098.950196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.248408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5099.282185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5103.380555] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5111.159076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.232616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5111.266341] Lustre: 212676:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5115.791860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5121.457708] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5125.455074] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5132.350869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5132.679735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5139.494893] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5144.383970] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5152.910443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5152.994973] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5158.847435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5163.540274] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5165.108901] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5168.589891] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5168.835388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5173.280065] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5173.528552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5177.183692] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ed83f58000.ost_server_uuid 50 [ 5178.710880] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88ed83f58000.ost_server_uuid in FULL state after 0 sec [ 5179.713150] Lustre: Permanently deactivating lustre-OST0001 [ 5179.725953] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5182.528131] Lustre: Permanently reactivating lustre-OST0001 [ 5189.097038] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5189.098725] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.102579] LustreError: Skipped 1 previous similar message [ 5189.115370] Lustre: Skipped 11 previous similar messages [ 5191.843478] Lustre: server umount lustre-OST0001 complete [ 5194.225165] LustreError: 214815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.257499] LustreError: 214815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 5201.549337] Lustre: server umount lustre-OST0000 complete [ 5211.409621] Lustre: server umount lustre-MDT0000 complete [ 5215.002051] LustreError: 212669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332036 with bad export cookie 1442656333289398013 [ 5215.013213] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5215.019101] LustreError: 212669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5215.418080] Lustre: server umount lustre-MDT0001 complete [ 5237.870367] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:40:57 (1780332057) [ 5244.651446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.917036] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5244.958350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5244.968082] Lustre: Skipped 5 previous similar messages [ 5249.037282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5257.248572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.312027] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5257.337968] Lustre: 218789:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5261.884841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5267.812995] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5271.906422] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5278.391327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5278.626550] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5278.661787] Lustre: Permanently deactivating lustre-OST0000 [ 5278.682150] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5284.713578] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5288.457942] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5289.209500] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5294.707676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5294.813748] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5296.033844] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5300.533290] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5306.150822] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5307.859609] Lustre: Permanently reactivating lustre-OST0000 [ 5307.903275] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5307.912755] Lustre: Skipped 2 previous similar messages [ 5316.681924] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5316.694612] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.709034] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5316.754460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.770708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5322.845703] Lustre: server umount lustre-OST0001 complete [ 5332.840493] Lustre: server umount lustre-OST0000 complete [ 5342.879344] Lustre: server umount lustre-MDT0000 complete [ 5346.131193] LustreError: 219517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332167 with bad export cookie 1442656333289399665 [ 5346.136901] LustreError: 219517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5346.139184] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.488774] Lustre: server umount lustre-MDT0001 complete [ 5355.937805] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5358.605357] Key type lgssc unregistered [ 5358.918907] LNet: 223198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5358.937441] LNetError: 223198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5358.964815] LNet: Removed LNI 192.168.204.146@tcp [ 5359.859169] Key type .llcrypt unregistered [ 5359.863069] Key type ._llcrypt unregistered [ 5381.261743] Key type ._llcrypt registered [ 5381.264954] Key type .llcrypt registered [ 5381.370066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5382.502318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5382.536945] alg: No test for adler32 (adler32-zlib) [ 5383.638963] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 5383.943799] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5385.624742] Key type lgssc registered [ 5386.805600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5396.274355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5396.307325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.943387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5402.660483] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5408.231822] LustreError: 224649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.249508] LustreError: 224649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5411.921973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.054677] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5412.088554] Lustre: 224637:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.097536] Lustre: Permanently deactivating lustre-MDT0001 [ 5412.110427] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5412.390758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.681602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5417.986613] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5420.836630] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.649374] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5429.056296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5429.438936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.173105] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5437.605810] LustreError: 226274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.982520] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.041868] LustreError: 226275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.102692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5449.186479] LustreError: 226273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.235127] LustreError: 226273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5450.204204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.505858] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5455.882676] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5455.894622] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5458.148446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5464.874773] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5468.962895] Lustre: Permanently reactivating lustre-MDT0001 [ 5468.995682] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5469.001672] Lustre: Skipped 2 previous similar messages [ 5472.765820] Lustre: lustre-MDT0001: Client 253e4da5-3690-42f7-8251-8970e5868987 (at 192.168.204.46@tcp) reconnecting [ 5476.779405] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5478.323296] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5479.004635] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.022840] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5479.038813] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5479.064954] LustreError: 227926:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5479.069631] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5479.077778] LustreError: 227926:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5479.077898] LustreError: 227926:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -108 [ 5479.077917] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5479.077917] [ 5479.080934] LustreError: 224640:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5483.473485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5483.679104] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5487.909699] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5488.141288] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5490.568245] Lustre: Permanently deactivating lustre-MDT0001 [ 5490.585867] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5490.590544] Lustre: Skipped 2 previous similar messages [ 5497.937869] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5507.044102] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.068571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.072335] Lustre: Skipped 3 previous similar messages [ 5507.086474] Lustre: Skipped 2 previous similar messages [ 5508.139848] Lustre: server umount lustre-MDT0000 complete [ 5512.052301] LustreError: 224628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332333 with bad export cookie 4835223563256676139 [ 5512.053863] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5512.065679] LustreError: 224628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5512.167094] LustreError: 228568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.179958] LustreError: 228568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5512.433272] Lustre: server umount lustre-MDT0001 complete [ 5516.002199] Lustre: server umount lustre-OST0000 complete [ 5528.145345] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:45:47 (1780332347) [ 5539.052326] Lustre: server umount lustre-OST0001 complete [ 5545.248862] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5554.741456] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5591.292226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.579641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5591.598903] Lustre: Skipped 2 previous similar messages [ 5591.632916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5591.761174] Lustre: lustre-MDT0000: new disk, initializing [ 5591.851922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.868982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5596.397345] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5607.213768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.403361] Lustre: 232312:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5607.501571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5607.506165] Lustre: Skipped 1 previous similar message [ 5607.575454] Lustre: lustre-MDT0001: new disk, initializing [ 5607.665245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5607.701861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5607.712977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5611.948680] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5621.525640] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5626.552771] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5634.567352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5634.909529] Lustre: lustre-OST0000: new disk, initializing [ 5634.913502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5634.924040] Lustre: 234055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5635.032749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.630337] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5645.390609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5645.410547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5645.550949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5649.549393] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5652.212795] LustreError: 232318:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5662.240895] LustreError: 232318:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5662.270369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.279270] LustreError: 232320:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5662.433214] Lustre: lustre-OST0001: new disk, initializing [ 5662.436820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5662.441328] Lustre: 235058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5667.856198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5672.056395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5672.066663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5672.151359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5672.359524] LustreError: 232320:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5675.579831] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5677.353490] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5680.146410] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 5680.152018] Lustre: Skipped 1 previous similar message [ 5682.673822] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.682268] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5682.688733] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5682.723895] Lustre: Skipped 1 previous similar message [ 5685.218207] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 5685.570218] Lustre: server umount lustre-OST0001 complete [ 5687.788204] LustreError: 235496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.811209] LustreError: 235496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5703.140176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.147805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.158016] Lustre: Skipped 2 previous similar messages [ 5703.177296] Lustre: Skipped 1 previous similar message [ 5708.261525] LustreError: 235496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.282178] LustreError: 235496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5709.181411] Lustre: server umount lustre-OST0000 complete [ 5713.376645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.377526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.385216] Lustre: Skipped 1 previous similar message [ 5713.407687] Lustre: Skipped 3 previous similar messages [ 5719.531375] Lustre: server umount lustre-MDT0000 complete [ 5723.085869] LustreError: 232305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332544 with bad export cookie 4835223563256678736 [ 5723.089093] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.092443] LustreError: 232305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5729.383775] Lustre: server umount lustre-MDT0001 complete [ 5739.483680] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5742.205832] Key type lgssc unregistered [ 5742.520541] LNet: 236962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5742.537630] LNetError: 236962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5742.564717] LNet: Removed LNI 192.168.204.146@tcp [ 5743.640730] Key type .llcrypt unregistered [ 5743.644862] Key type ._llcrypt unregistered [ 5779.125288] Key type ._llcrypt registered [ 5779.127211] Key type .llcrypt registered [ 5779.247449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5780.270503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5780.313743] alg: No test for adler32 (adler32-zlib) [ 5781.376474] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 5781.646823] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5783.311224] Key type lgssc registered [ 5784.430323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5792.295549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5792.317716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.576344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5793.596389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5793.801933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.587417] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5805.664982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.818279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5805.876324] Lustre: 239152:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5806.183073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5811.699414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5819.064273] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5824.565787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5832.390218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.587374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5832.789367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.139663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5839.895842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5846.013856] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5900.776387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.782131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5900.794454] Lustre: Skipped 1 previous similar message [ 5900.798422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5900.806660] Lustre: Skipped 1 previous similar message [ 5903.998557] Lustre: server umount lustre-OST0000 complete [ 5908.823829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5920.253596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5920.574375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5921.591902] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.628349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.631431] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5926.344395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5932.971756] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5972.966341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5972.981935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.017653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.125725] Lustre: server umount lustre-OST0000 complete [ 5983.201273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.206387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.221505] Lustre: Skipped 2 previous similar messages [ 5983.250105] Lustre: Skipped 2 previous similar messages [ 5988.003403] Lustre: server umount lustre-MDT0000 complete [ 5991.983189] LustreError: 239145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332813 with bad export cookie 4109687028771486276 [ 5991.987283] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.994383] LustreError: 239145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5992.411806] Lustre: server umount lustre-MDT0001 complete [ 6002.740424] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6005.580180] Key type lgssc unregistered [ 6005.820839] LNet: 244110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6005.830758] LNetError: 244110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.842466] LNet: Removed LNI 192.168.204.146@tcp [ 6007.049604] Key type .llcrypt unregistered [ 6007.056301] Key type ._llcrypt unregistered [ 6027.631154] Key type ._llcrypt registered [ 6027.633087] Key type .llcrypt registered [ 6027.772376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6028.590696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6028.694140] alg: No test for adler32 (adler32-zlib) [ 6029.708344] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6029.888234] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6031.551774] Key type lgssc registered [ 6032.570191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.957733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6038.974986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.413215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.470359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6050.792672] LustreError: 245488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.809944] LustreError: 245488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6052.491726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.756723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6057.025371] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6062.616294] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6066.174099] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6072.490599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.856637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6079.095204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6083.720233] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6085.254739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6108.135924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6108.145316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.162628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.215592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.224035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.242611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.269026] Lustre: Skipped 1 previous similar message [ 6113.086343] Lustre: server umount lustre-OST0000 complete [ 6119.395319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6119.400490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.430200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.440222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.442263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.735541] Lustre: server umount lustre-MDT0000 complete [ 6126.561833] LustreError: 245484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.589119] LustreError: 245484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6128.661598] LustreError: 245467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332949 with bad export cookie 17623792958914900552 [ 6128.675661] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6128.925095] Lustre: server umount lustre-MDT0001 complete [ 6139.348805] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6142.413512] Key type lgssc unregistered [ 6142.756524] LNet: 249439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6142.779384] LNetError: 249439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6142.796075] LNet: Removed LNI 192.168.204.146@tcp [ 6143.503136] Key type .llcrypt unregistered [ 6143.504948] Key type ._llcrypt unregistered [ 6159.705643] Key type ._llcrypt registered [ 6159.710185] Key type .llcrypt registered [ 6159.885264] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6160.823683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6160.839184] alg: No test for adler32 (adler32-zlib) [ 6161.921334] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6162.136318] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6163.823297] Key type lgssc registered [ 6164.909088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6172.211252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6172.236943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.768835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.964203] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6182.882469] LustreError: 250648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.902138] LustreError: 250648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6186.169576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.544408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.882864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6196.022963] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6199.953565] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6207.010944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6207.436059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.431847] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6218.744616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6218.908478] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6232.032739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6232.041882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.057814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.854396] Lustre: server umount lustre-OST0000 complete [ 6237.664894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6237.677721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.697487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.200717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.202141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.240847] Lustre: Skipped 1 previous similar message [ 6244.322024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6244.328208] Lustre: Skipped 1 previous similar message [ 6246.412258] Lustre: server umount lustre-MDT0000 complete [ 6249.441562] LustreError: 250649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.457871] LustreError: 250649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6249.796064] LustreError: 250629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333070 with bad export cookie 15263501977869062590 [ 6249.804132] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6250.055501] Lustre: server umount lustre-MDT0001 complete [ 6259.348225] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6261.997188] Key type lgssc unregistered [ 6262.312604] LNet: 253969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6262.326295] LNetError: 253969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6262.345894] LNet: Removed LNI 192.168.204.146@tcp [ 6263.391236] Key type .llcrypt unregistered [ 6263.394381] Key type ._llcrypt unregistered [ 6288.658801] Key type ._llcrypt registered [ 6288.661711] Key type .llcrypt registered [ 6288.795589] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6289.881370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6289.977899] alg: No test for adler32 (adler32-zlib) [ 6291.149567] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6291.350948] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6293.079959] Key type lgssc registered [ 6294.326441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.872719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.889296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.475435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.852367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6313.958248] LustreError: 255371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.986956] LustreError: 255371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6317.096103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.414223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6321.476084] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6326.677501] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6330.585308] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.980780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.412691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.702334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6347.693082] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6350.769569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6376.426161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.427327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.450427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6376.473121] Lustre: Skipped 1 previous similar message [ 6381.119932] Lustre: server umount lustre-OST0000 complete [ 6386.656647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.682427] Lustre: Skipped 1 previous similar message [ 6386.696869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6386.707022] Lustre: Skipped 1 previous similar message [ 6391.104545] Lustre: server umount lustre-MDT0000 complete [ 6391.781332] LustreError: 255371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.816120] LustreError: 255371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6394.656773] LustreError: 255352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333215 with bad export cookie 770667244923704145 [ 6394.674541] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6395.118057] Lustre: server umount lustre-MDT0001 complete [ 6406.600662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6409.330481] Key type lgssc unregistered [ 6409.636708] LNet: 259319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6409.643189] LNetError: 259319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6409.662233] LNet: Removed LNI 192.168.204.146@tcp [ 6410.624925] Key type .llcrypt unregistered [ 6410.626637] Key type ._llcrypt unregistered [ 6430.772444] Key type ._llcrypt registered [ 6430.775249] Key type .llcrypt registered [ 6430.911458] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6432.259824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6432.292544] alg: No test for adler32 (adler32-zlib) [ 6433.339809] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6433.639519] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6435.375576] Key type lgssc registered [ 6436.764813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6445.191780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6445.223408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.998726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.284806] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6457.315402] LustreError: 260551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.344422] LustreError: 260551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6460.172147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.801436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6465.451471] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6471.307428] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6476.188144] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6482.083726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.412368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.740956] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6492.476627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6492.676335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6502.880659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.884639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.891487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.897526] Lustre: Skipped 1 previous similar message [ 6502.914793] Lustre: Skipped 1 previous similar message [ 6508.001212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6508.007076] Lustre: Skipped 1 previous similar message [ 6509.012278] Lustre: server umount lustre-OST0000 complete [ 6512.099448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6512.108574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6512.121580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.192644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.202616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.752893] Lustre: server umount lustre-MDT0000 complete [ 6521.220756] LustreError: 262962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333342 with bad export cookie 14580860815677712165 [ 6521.226388] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.244290] LustreError: 262962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6521.552456] Lustre: server umount lustre-MDT0001 complete [ 6530.449943] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6532.680957] Key type lgssc unregistered [ 6532.986633] LNet: 263899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6533.002585] LNetError: 263899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6533.015325] LNet: Removed LNI 192.168.204.146@tcp [ 6533.698298] Key type .llcrypt unregistered [ 6533.702721] Key type ._llcrypt unregistered [ 6543.202534] Key type ._llcrypt registered [ 6543.205735] Key type .llcrypt registered [ 6543.343724] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6556.160681] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6567.726442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6568.278644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6568.333817] alg: No test for adler32 (adler32-zlib) [ 6569.355297] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6569.497930] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6571.135176] Key type lgssc registered [ 6572.068247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6600.664632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6600.754591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.017466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6602.081330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6602.199749] Lustre: lustre-MDT0000: new disk, initializing [ 6602.286725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.322156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6605.837571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6615.182141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.248961] Lustre: 267378:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6615.301680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6615.304908] Lustre: Skipped 1 previous similar message [ 6615.375759] Lustre: lustre-MDT0001: new disk, initializing [ 6615.445729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.484104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6615.497759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6619.458808] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6629.577618] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6633.430328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6639.272796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.510854] Lustre: lustre-OST0000: new disk, initializing [ 6639.519797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6639.531678] Lustre: 269124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6639.616949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.241533] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6648.403815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6648.426344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6648.578201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6652.627321] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6657.267165] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6657.470355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6661.697149] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6661.860535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.164648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6664.165673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.183535] LustreError: Skipped 1 previous similar message [ 6664.187535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.211818] Lustre: Skipped 1 previous similar message [ 6669.280844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.285239] Lustre: Skipped 2 previous similar messages [ 6669.787270] Lustre: server umount lustre-OST0000 complete [ 6674.416274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.417424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.439843] Lustre: Skipped 1 previous similar message [ 6674.450597] Lustre: Skipped 1 previous similar message [ 6679.522923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.533385] Lustre: Skipped 1 previous similar message [ 6684.640624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6684.654075] Lustre: Skipped 1 previous similar message [ 6687.199120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6687.375812] Lustre: server umount lustre-MDT0000 complete [ 6690.784541] LustreError: 267370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333511 with bad export cookie 16243490791967241143 [ 6690.786907] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.806962] LustreError: 267370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6691.207438] Lustre: server umount lustre-MDT0001 complete [ 6698.916418] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:05:18 (1780333518) [ 6704.655604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.048023] LustreError: 271389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.064230] LustreError: 271389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6705.152148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6708.613742] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6710.241308] LustreError: 271390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.362149] LustreError: 271389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.131039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.494902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6720.335703] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6724.731747] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6727.887231] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6732.494607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6732.741441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.268677] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6741.832164] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6761.444251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6761.459996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.484183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.542285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.545249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.744939] Lustre: server umount lustre-OST0000 complete [ 6770.658643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.123154] Lustre: server umount lustre-MDT0000 complete [ 6779.259883] LustreError: 271370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333600 with bad export cookie 16243490791967242067 [ 6779.264437] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.271857] LustreError: 271370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6779.640146] Lustre: server umount lustre-MDT0001 complete [ 6789.295656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6792.596565] Key type lgssc unregistered [ 6792.942581] LNet: 274589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6792.964220] LNetError: 274589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6794.026224] LNet: Removed LNI 192.168.204.146@tcp [ 6795.130510] Key type .llcrypt unregistered [ 6795.132758] Key type ._llcrypt unregistered [ 6823.026782] Key type ._llcrypt registered [ 6823.030429] Key type .llcrypt registered [ 6823.149227] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6837.896570] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6838.784706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6838.871024] alg: No test for adler32 (adler32-zlib) [ 6839.905840] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 6840.151768] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6841.825203] Key type lgssc registered [ 6842.936407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6889.015272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6889.056393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.458434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6890.519887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6890.683868] Lustre: lustre-MDT0000: new disk, initializing [ 6890.787624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.805580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6895.205969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6905.889586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.047367] Lustre: 278387:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6906.114786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6906.120130] Lustre: Skipped 1 previous similar message [ 6906.205808] Lustre: lustre-MDT0001: new disk, initializing [ 6906.322142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.362027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6906.388412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6911.236438] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6918.025839] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6921.629628] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6930.330773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.627250] Lustre: lustre-OST2710: new disk, initializing [ 6930.630957] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6930.639312] Lustre: 280127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6930.731046] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6937.438314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6939.204256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6939.213922] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6939.317824] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6945.176335] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6952.664836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.820235] Lustre: lustre-OST03e8: new disk, initializing [ 6952.823595] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6952.828837] Lustre: 281080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6952.910464] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6958.729366] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6961.749981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6961.765612] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6961.856977] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6964.071496] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6971.263983] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6971.505348] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6975.758610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6976.087801] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6980.209702] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6980.388859] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6984.631630] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6984.828307] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6992.867988] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.884359] Lustre: Skipped 2 previous similar messages [ 6992.887694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6992.891085] Lustre: Skipped 3 previous similar messages [ 6994.916434] Lustre: server umount lustre-MDT0000 complete [ 6997.768260] LustreError: 278379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333818 with bad export cookie 15855155559994599105 [ 6997.771906] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.774547] LustreError: 278379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6997.963975] Lustre: server umount lustre-MDT0001 complete [ 7012.078880] Lustre: server umount lustre-OST2710 complete [ 7014.367329] Lustre: 276182:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780333819/real 1780333819] req@00000000d79c8740 x1866815148984064/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780333835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7014.401163] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.427146] Lustre: Skipped 1 previous similar message [ 7016.287203] Lustre: server umount lustre-OST03e8 complete [ 7036.799533] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7045.417734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7079.208584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.476601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7079.511557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7079.586398] Lustre: lustre-MDT0000: new disk, initializing [ 7079.675540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.698586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7083.839639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7093.457617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.582675] Lustre: 286109:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7093.642636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7093.646938] Lustre: Skipped 1 previous similar message [ 7093.740695] Lustre: lustre-MDT0001: new disk, initializing [ 7093.846813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.914088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7093.933992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7098.030944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7106.836155] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7110.946435] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.814406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.150885] Lustre: lustre-OST0000: new disk, initializing [ 7119.155252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7119.162592] Lustre: 287853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7119.310167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.575449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7124.606860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7124.710017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7126.125221] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7133.452731] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7138.281714] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7138.445771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7143.996889] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7144.348195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7150.566897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7150.572049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.581860] LustreError: Skipped 1 previous similar message [ 7150.604607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7150.618892] Lustre: Skipped 1 previous similar message [ 7152.550574] Lustre: server umount lustre-OST0000 complete [ 7160.290560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7160.298688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.318563] Lustre: Skipped 1 previous similar message [ 7160.327964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.095477] Lustre: server umount lustre-MDT0000 complete [ 7169.439344] LustreError: 286103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333990 with bad export cookie 15855155559994600820 [ 7169.441879] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7169.453338] LustreError: 286103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7169.854415] Lustre: server umount lustre-MDT0001 complete [ 7180.465321] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:13:19 (1780333999) [ 7182.169073] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7184.343942] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:13:23 (1780334003) [ 7188.654435] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7221.279704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.550552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7221.587758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7221.787238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.513716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7235.328522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.522227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7235.554561] Lustre: 291550:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7240.060704] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7262.708946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.940136] Lustre: Denying initial registration attempt from nid 192.168.204.146@tcp, specified as failover [ 7262.953567] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7262.968980] LustreError: 292971:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -99 [ 7262.979868] LustreError: 292971:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7262.990869] LustreError: 292971:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7263.080148] Lustre: server umount lustre-OST0000 complete [ 7263.084691] LustreError: 292971:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7271.909744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.923472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.944496] Lustre: Skipped 1 previous similar message [ 7271.947688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.959588] Lustre: Skipped 1 previous similar message [ 7279.071894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7279.078037] Lustre: Skipped 3 previous similar messages [ 7286.239271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7286.564795] Lustre: server umount lustre-MDT0000 complete [ 7289.312668] LustreError: 292292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.332770] LustreError: 292292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7290.323584] LustreError: 291541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334111 with bad export cookie 15855155559994601751 [ 7290.332436] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7290.344053] LustreError: 291541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7290.655970] Lustre: server umount lustre-MDT0001 complete [ 7301.606668] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7303.889478] Key type lgssc unregistered [ 7304.185264] LNet: 294077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7304.189819] LNetError: 294077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7304.202244] LNet: Removed LNI 192.168.204.146@tcp [ 7304.881417] Key type .llcrypt unregistered [ 7304.883585] Key type ._llcrypt unregistered [ 7316.121769] Key type ._llcrypt registered [ 7316.124661] Key type .llcrypt registered [ 7316.256134] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7317.295335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.354055] alg: No test for adler32 (adler32-zlib) [ 7318.368451] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 7318.687425] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7320.379848] Key type lgssc registered [ 7321.653589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.510473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7377.998408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7378.036013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.260592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7379.278594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7379.408792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.863610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7392.419786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.543178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7392.607259] Lustre: 296809:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7392.927679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.779866] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7403.431755] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7407.093987] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7426.541741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.770355] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7426.935452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.937237] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7437.185169] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7443.425037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7443.435493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.440810] LustreError: Skipped 1 previous similar message [ 7443.459774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7445.226171] Lustre: server umount lustre-OST0000 complete [ 7448.546954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.550909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7448.560314] Lustre: Skipped 2 previous similar messages [ 7448.574570] Lustre: Skipped 1 previous similar message [ 7453.665794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.673225] Lustre: Skipped 2 previous similar messages [ 7454.727078] Lustre: server umount lustre-MDT0000 complete [ 7458.362277] LustreError: 296799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334279 with bad export cookie 16051641474880774547 [ 7458.372708] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7458.379257] LustreError: 296799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7458.845592] Lustre: server umount lustre-MDT0001 complete [ 7469.306740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7472.843714] Key type lgssc unregistered [ 7473.325924] LNet: 300024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7473.330787] LNetError: 300024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7473.352101] LNet: Removed LNI 192.168.204.146@tcp [ 7474.303341] Key type .llcrypt unregistered [ 7474.306140] Key type ._llcrypt unregistered [ 7499.720120] Key type ._llcrypt registered [ 7499.721441] Key type .llcrypt registered [ 7499.846640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7501.222860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7501.245547] alg: No test for adler32 (adler32-zlib) [ 7502.291608] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 7502.520186] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7504.159175] Key type lgssc registered [ 7505.376453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7512.741612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7512.761173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.204638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.456618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7524.329639] LustreError: 301450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.339913] LustreError: 301450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7526.741438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.096047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7531.011791] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7537.452981] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7541.511805] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7550.009382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.422227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7556.303819] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7562.050175] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7569.892849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.900741] Lustre: Skipped 1 previous similar message [ 7569.904269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7569.910674] Lustre: Skipped 2 previous similar messages [ 7574.008786] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 7575.011409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.025568] Lustre: Skipped 1 previous similar message [ 7575.251765] Lustre: server umount lustre-MDT0000 complete [ 7579.103989] LustreError: 301447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.231151] LustreError: 301432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334400 with bad export cookie 7876005312663981439 [ 7579.231996] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.242537] LustreError: 301432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7580.137675] LustreError: 302856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.139632] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.186087] Lustre: Skipped 1 previous similar message [ 7580.195577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7584.161765] LustreError: 303466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.247930] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 7584.255758] Lustre: Skipped 1 previous similar message [ 7585.663353] Lustre: server umount lustre-MDT0001 complete [ 7592.037755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7600.214501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.027374] LustreError: 303057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.043297] LustreError: 303057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7605.143283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.217988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7609.798300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7614.402459] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7614.403628] LustreError: 304705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.437062] LustreError: 304705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7619.532374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.099569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7625.194569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7625.298649] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7631.284758] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7632.867460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7636.653868] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7638.143808] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7645.664892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7645.675881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.682411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.692542] Lustre: Skipped 1 previous similar message [ 7645.712019] Lustre: Skipped 1 previous similar message [ 7649.635471] Lustre: server umount lustre-OST0000 complete [ 7655.904551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.932698] Lustre: Skipped 1 previous similar message [ 7661.039195] LustreError: 304711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.076972] LustreError: 304711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7661.265447] Lustre: server umount lustre-MDT0000 complete [ 7665.680791] LustreError: 301431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334486 with bad export cookie 7876005312663989118 [ 7665.695231] LustreError: 301431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7665.695515] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7666.209069] Lustre: server umount lustre-MDT0001 complete [ 7677.773417] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7681.287784] Key type lgssc unregistered [ 7681.584069] LNet: 307137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7681.590375] LNetError: 307137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7681.607332] LNet: Removed LNI 192.168.204.146@tcp [ 7682.437532] Key type .llcrypt unregistered [ 7682.440165] Key type ._llcrypt unregistered [ 7705.797695] Key type ._llcrypt registered [ 7705.800824] Key type .llcrypt registered [ 7705.928927] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7706.565858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7706.672668] alg: No test for adler32 (adler32-zlib) [ 7707.731425] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 7707.954852] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7709.623486] Key type lgssc registered [ 7710.583841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7717.409565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7717.432645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.919144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.471400] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7729.127675] LustreError: 308541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.152137] LustreError: 308541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7732.012694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.370604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.003992] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7742.301360] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7745.992425] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7752.679231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.687811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.701213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7754.210624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7754.214701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.478564] Lustre: server umount lustre-MDT0000 complete [ 7757.355990] LustreError: 309251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.386031] LustreError: 309251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7758.944085] LustreError: 308521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334580 with bad export cookie 2186859577767769596 [ 7758.961821] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7758.963534] LustreError: 308521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7759.235736] Lustre: server umount lustre-MDT0001 complete [ 7772.925843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.237077] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7773.299353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7773.540446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.558264] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7786.795387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.922253] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7786.979121] Lustre: 310902:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7787.303042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7791.577232] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7797.806992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.801684] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7811.825585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7812.079129] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7812.089979] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7812.103412] LustreError: 312560:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -2 [ 7812.113906] LustreError: 312560:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7812.119724] LustreError: 312560:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7812.205317] Lustre: server umount lustre-OST0000 complete [ 7812.219399] LustreError: 312560:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7822.505497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.788350] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7822.951515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.077766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7830.679597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7834.626741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7841.550943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7841.635699] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7841.709567] Lustre: lustre-OST0001: new disk, initializing [ 7841.712876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7841.717536] Lustre: 313991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7841.786552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7847.036666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7847.045549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7847.157157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7847.499884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7852.684113] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7857.632539] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7857.648304] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.657379] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7857.661036] Lustre: Skipped 1 previous similar message [ 7857.682636] Lustre: Skipped 2 previous similar messages [ 7861.552436] Lustre: server umount lustre-OST0001 complete [ 7862.771098] LustreError: 313029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7862.805568] LustreError: 313029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7866.353293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7866.361044] LustreError: Skipped 1 previous similar message [ 7866.364315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.381518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7867.874946] LustreError: 313029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.912275] LustreError: 313029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7871.859518] Lustre: server umount lustre-OST0000 complete [ 7878.117654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.130427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.135550] Lustre: Skipped 2 previous similar messages [ 7878.151854] Lustre: Skipped 4 previous similar messages [ 7882.009430] Lustre: server umount lustre-MDT0000 complete [ 7883.235288] LustreError: 310913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.250915] LustreError: 310913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7886.154460] LustreError: 310894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334707 with bad export cookie 2186859577767770149 [ 7886.156273] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.170133] LustreError: 310894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7886.581040] Lustre: server umount lustre-MDT0001 complete [ 7896.847102] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7899.735446] Key type lgssc unregistered [ 7900.118524] LNet: 315818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7900.127827] LNetError: 315818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7900.163705] LNet: Removed LNI 192.168.204.146@tcp [ 7901.019591] Key type .llcrypt unregistered [ 7901.021878] Key type ._llcrypt unregistered [ 7971.953088] Key type ._llcrypt registered [ 7971.954949] Key type .llcrypt registered [ 7972.094739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7987.218652] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7988.317195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7988.430264] alg: No test for adler32 (adler32-zlib) [ 7989.518938] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 7989.797847] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7991.463388] Key type lgssc registered [ 7992.665646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8027.175929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8027.195379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.419112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8028.466952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8028.561995] Lustre: lustre-MDT0000: new disk, initializing [ 8028.662727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.691645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8032.931484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8043.502072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.619564] Lustre: 320717:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8043.682692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8043.687405] Lustre: Skipped 1 previous similar message [ 8043.747603] Lustre: lustre-MDT0001: new disk, initializing [ 8043.848672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8043.881519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8043.897045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8048.608943] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8056.529381] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8061.005513] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8067.172937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8067.351223] Lustre: lustre-OST0000: new disk, initializing [ 8067.354362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8067.362773] Lustre: 322464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8067.445352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.076695] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8074.269454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8074.281993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8074.412787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8079.540125] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8084.651056] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8084.908979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8089.355374] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8089.604993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8094.693315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.715153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.736138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8095.202438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8097.555886] Lustre: server umount lustre-OST0000 complete [ 8105.448897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.449228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8105.470420] Lustre: Skipped 1 previous similar message [ 8105.477285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.477293] Lustre: Skipped 2 previous similar messages [ 8108.004681] Lustre: server umount lustre-MDT0000 complete [ 8111.903743] LustreError: 320712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334933 with bad export cookie 2365353565710169854 [ 8111.911186] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.916181] LustreError: 320712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8112.329576] Lustre: server umount lustre-MDT0001 complete [ 8120.323429] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:29:00 (1780334940) [ 8128.171824] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:29:07 (1780334947) [ 8138.763330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.586992] LustreError: 325074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.605019] LustreError: 325074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8139.678487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.889512] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8144.868827] LustreError: 325075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8148.974565] LustreError: 325074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.759477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.118711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8155.765318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8161.381062] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8165.754992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8172.817754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8173.382168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8179.449359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8183.725471] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8196.277683] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8198.650620] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8200.635865] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8202.575071] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8204.729208] Lustre: Failing over lustre-MDT0000 [ 8204.797336] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 8204.808992] Lustre: Skipped 1 previous similar message [ 8205.174774] Lustre: server umount lustre-MDT0000 complete [ 8205.287469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.301608] LustreError: 325074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.312776] Lustre: Skipped 1 previous similar message [ 8205.351788] LustreError: 325074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8209.881753] LustreError: 325070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.121458] LustreError: 326235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.147047] LustreError: 326235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8221.663087] Lustre: 318944:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780335026/real 1780335026] req@0000000053af3443 x1866816353803776/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780335042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8221.699807] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8224.936165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.138548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.191732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8234.968860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8235.489543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8236.533032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8236.558701] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8236.613258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8242.588302] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8244.161403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8247.310472] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8248.983314] Lustre: Failing over lustre-MDT0000 [ 8249.225893] Lustre: server umount lustre-MDT0000 complete [ 8250.885218] LustreError: 326235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.897445] LustreError: 326235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8251.873878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.880178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8251.882319] Lustre: Skipped 2 previous similar messages [ 8266.640120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.745424] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8266.973746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.018479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8267.141193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8270.709792] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8272.362447] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8272.367848] Lustre: Skipped 2 previous similar messages [ 8272.393707] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8272.444415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8277.540499] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8279.061243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8287.721163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.729434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8287.753081] Lustre: Skipped 3 previous similar messages [ 8287.759857] Lustre: Skipped 1 previous similar message [ 8291.277562] Lustre: server umount lustre-MDT0000 complete [ 8292.833560] LustreError: 325797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.857913] LustreError: 325797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8294.793970] LustreError: 325054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335115 with bad export cookie 2365353565710172605 [ 8294.799056] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.808390] LustreError: 325054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8295.212095] Lustre: server umount lustre-MDT0001 complete [ 8309.561787] Lustre: server umount lustre-OST0000 complete [ 8323.507436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.301425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8335.683029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.032166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8336.039213] Lustre: Skipped 1 previous similar message [ 8340.112955] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8345.067467] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8347.778664] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8353.158390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8358.905459] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8363.791848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8368.883230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8374.242593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.271527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.272574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.276297] Lustre: Skipped 2 previous similar messages [ 8376.041819] Lustre: server umount lustre-OST0000 complete [ 8384.480475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8384.486502] Lustre: Skipped 3 previous similar messages [ 8385.583748] Lustre: server umount lustre-MDT0000 complete [ 8388.924618] LustreError: 330856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335210 with bad export cookie 2365353565710173354 [ 8388.929298] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8388.936790] LustreError: 330856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8389.212752] Lustre: server umount lustre-MDT0001 complete [ 8399.286785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8402.069738] Key type lgssc unregistered [ 8402.381162] LNet: 334052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8402.392282] LNetError: 334052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8402.407216] LNet: Removed LNI 192.168.204.146@tcp [ 8403.152233] Key type .llcrypt unregistered [ 8403.153884] Key type ._llcrypt unregistered [ 8429.132703] Key type ._llcrypt registered [ 8429.136579] Key type .llcrypt registered [ 8429.292436] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8430.591315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8430.743222] alg: No test for adler32 (adler32-zlib) [ 8431.922680] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 8432.216893] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8434.015188] Key type lgssc registered [ 8435.053253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8442.922033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8442.947124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.640112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.600720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8454.116798] LustreError: 335573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.138430] LustreError: 335573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8458.180683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.674157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8463.558900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8469.954250] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8474.003993] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8480.820459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.021648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.864482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8489.215613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8493.497217] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8504.805822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.817855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.820115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8504.830216] Lustre: Skipped 2 previous similar messages [ 8504.840981] Lustre: Skipped 1 previous similar message [ 8506.806037] Lustre: server umount lustre-MDT0000 complete [ 8509.924344] LustreError: 337297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8509.948180] LustreError: 337297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8510.832559] LustreError: 335552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335332 with bad export cookie 6084913806449590245 [ 8510.843523] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8510.846152] LustreError: 335552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8511.190123] Lustre: server umount lustre-MDT0001 complete [ 8525.742535] Lustre: server umount lustre-OST0000 complete [ 8541.084666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.591819] LustreError: 338942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.701667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.193913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8546.793500] LustreError: 338943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.883486] LustreError: 338942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.541591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.969824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.273561] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8567.696855] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8571.744934] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8579.202640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.521094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.916431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8586.679808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8591.938320] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8607.205755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8607.207580] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.210266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8607.210274] Lustre: Skipped 1 previous similar message [ 8607.255791] Lustre: Skipped 2 previous similar messages [ 8610.282968] Lustre: server umount lustre-MDT0000 complete [ 8612.326903] LustreError: 338942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.347403] LustreError: 338942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8613.545502] LustreError: 339660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335434 with bad export cookie 6084913806449591351 [ 8613.553755] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.563672] LustreError: 339660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8613.904408] Lustre: server umount lustre-MDT0001 complete [ 8627.124369] Lustre: server umount lustre-OST0000 complete [ 8642.110587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.843048] LustreError: 342514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.927120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.622866] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8656.368811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.410049] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8667.274727] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8671.128625] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8678.167232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.601552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.606514] Lustre: Skipped 1 previous similar message [ 8685.499048] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8687.861291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8691.238490] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8697.312688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8697.330895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.358822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.372902] Lustre: Skipped 2 previous similar messages [ 8701.949253] Lustre: server umount lustre-OST0000 complete [ 8708.066539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8708.074098] LustreError: Skipped 1 previous similar message [ 8708.078718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8708.097657] Lustre: Skipped 1 previous similar message [ 8708.103489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.111572] Lustre: Skipped 2 previous similar messages [ 8713.697288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.709303] Lustre: Skipped 2 previous similar messages [ 8720.351130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8720.508910] Lustre: server umount lustre-MDT0000 complete [ 8722.913353] LustreError: 342511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.940602] LustreError: 342511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8724.839898] LustreError: 342495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335546 with bad export cookie 6084913806449592380 [ 8724.844552] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.850505] LustreError: 342495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8725.176523] Lustre: server umount lustre-MDT0001 complete [ 8737.198686] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8739.931917] Key type lgssc unregistered [ 8740.214517] LNet: 345690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8740.224441] LNetError: 345690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8740.237472] LNet: Removed LNI 192.168.204.146@tcp [ 8740.963181] Key type .llcrypt unregistered [ 8740.966292] Key type ._llcrypt unregistered [ 8767.852572] Key type ._llcrypt registered [ 8767.853848] Key type .llcrypt registered [ 8767.936843] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8768.767555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8768.821024] alg: No test for adler32 (adler32-zlib) [ 8770.037895] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 8770.357681] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8772.071468] Key type lgssc registered [ 8773.020161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8781.481514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8781.512109] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.528258] LustreError: 347118:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8781.570331] LustreError: 347118:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8781.587316] LustreError: 347118:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8781.595924] LustreError: 347118:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8781.605522] LustreError: 347118:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8789.245811] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8789.251043] LustreError: 347516:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8789.269433] LustreError: 347516:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8789.275916] LustreError: 347516:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8789.282368] LustreError: 347516:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8789.288870] LustreError: 347516:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8807.303629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8809.969535] Key type lgssc unregistered [ 8810.282725] LNet: 348501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8810.293408] LNetError: 348501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8810.313348] LNet: Removed LNI 192.168.204.146@tcp [ 8811.168644] Key type .llcrypt unregistered [ 8811.172731] Key type ._llcrypt unregistered [ 8829.387309] Key type ._llcrypt registered [ 8829.394791] Key type .llcrypt registered [ 8829.488661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8845.070837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8846.040996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.075390] alg: No test for adler32 (adler32-zlib) [ 8847.204313] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 8847.495582] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8849.207571] Key type lgssc registered [ 8850.429963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8886.396860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8886.483918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.856586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8887.895434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8888.021533] Lustre: lustre-MDT0000: new disk, initializing [ 8888.127658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8888.182206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8892.759701] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8905.710415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.037919] Lustre: 351627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8906.097464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8906.102186] Lustre: Skipped 1 previous similar message [ 8906.166505] Lustre: lustre-MDT0001: new disk, initializing [ 8906.359972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8906.411918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8906.424426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8910.510689] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8918.011170] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8921.227906] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8928.005640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.382608] Lustre: lustre-OST0000: new disk, initializing [ 8928.388677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8928.398924] Lustre: 353370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8928.498353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.814493] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8937.530637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8937.541551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8937.661110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8940.559449] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8944.771581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8944.948951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8949.440924] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8949.596699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8953.312605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8953.321875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.339362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8953.384941] Lustre: Skipped 1 previous similar message [ 8957.345848] Lustre: server umount lustre-OST0000 complete [ 8962.529542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8962.538316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.547059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8962.550750] Lustre: Skipped 1 previous similar message [ 8963.555045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.555731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.604671] Lustre: Skipped 1 previous similar message [ 8968.522837] Lustre: server umount lustre-MDT0000 complete [ 8972.051865] LustreError: 353369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335793 with bad export cookie 9171407531468731868 [ 8972.060985] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8972.073275] LustreError: 353369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8972.504981] Lustre: server umount lustre-MDT0001 complete [ 8982.328924] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8983.838392] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:43:23 (1780335803) [ 8990.190850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.711026] LustreError: 355681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8990.733282] LustreError: 355681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8990.804756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.247971] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8995.808650] LustreError: 355682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.910853] LustreError: 355681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.360303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.737177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.291113] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9013.100464] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9016.447777] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9022.409990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9022.803794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.304597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9032.984744] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9041.089107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9041.377525] Lustre: lustre-OST0001: new disk, initializing [ 9041.383597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9041.392288] Lustre: 358178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9041.520970] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9048.587527] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9051.289408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9051.307914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9051.416270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9055.653774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9061.858053] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9061.860804] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.864255] LustreError: Skipped 1 previous similar message [ 9061.868930] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9061.878916] Lustre: Skipped 1 previous similar message [ 9061.887817] Lustre: Skipped 1 previous similar message [ 9066.166178] Lustre: server umount lustre-OST0001 complete [ 9066.984388] LustreError: 357279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.012443] LustreError: 357279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9072.097364] LustreError: 357676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.137549] LustreError: 357676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9082.344494] LustreError: 357279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.350636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.361747] LustreError: 357279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9082.369155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9082.381874] Lustre: Skipped 1 previous similar message [ 9086.432898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9088.564819] Lustre: server umount lustre-OST0000 complete [ 9092.577472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.585906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9092.604167] Lustre: Skipped 2 previous similar messages [ 9092.623182] Lustre: Skipped 4 previous similar messages [ 9106.401743] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9106.608522] Lustre: server umount lustre-MDT0000 complete [ 9107.937278] LustreError: 356393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.953354] LustreError: 356393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9110.106631] LustreError: 355663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335931 with bad export cookie 9171407531468732799 [ 9110.115370] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9110.119615] LustreError: 355663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9116.570574] Lustre: server umount lustre-MDT0001 complete [ 9125.966781] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9128.332521] Key type lgssc unregistered [ 9128.593580] LNet: 360035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9128.599600] LNetError: 360035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9128.612600] LNet: Removed LNI 192.168.204.146@tcp [ 9129.224493] Key type .llcrypt unregistered [ 9129.228744] Key type ._llcrypt unregistered [ 9182.157291] Key type ._llcrypt registered [ 9182.159428] Key type .llcrypt registered [ 9182.264430] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9183.354603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9183.382135] alg: No test for adler32 (adler32-zlib) [ 9184.465870] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 9184.669819] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9186.327178] Key type lgssc registered [ 9187.475335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9195.231785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9195.270143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.631884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9196.650915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9196.785172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9200.523428] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9208.393327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9208.472767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9208.508395] Lustre: 362839:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9208.746306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9213.092030] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9218.798782] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9222.696514] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9228.780546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.793374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.800857] Lustre: Skipped 1 previous similar message [ 9228.823958] Lustre: Skipped 1 previous similar message [ 9230.902145] Lustre: server umount lustre-MDT0000 complete [ 9234.248280] LustreError: 362830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336055 with bad export cookie 13230564568306568348 [ 9234.259365] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.264922] LustreError: 362830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9234.608584] Lustre: server umount lustre-MDT0001 complete [ 9242.795888] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:47:42 (1780336062) [ 9249.027890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.486974] LustreError: 365126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.496302] LustreError: 365126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9249.554891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9253.552563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9254.882184] LustreError: 365127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9260.007513] LustreError: 365126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9261.867532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.143443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9266.475728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9271.972609] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9276.327529] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9284.852492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.137529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9285.439522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.558658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9298.522717] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9307.546820] Lustre: Permanently deactivating lustre-OST0000 [ 9307.574507] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9316.864146] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9319.709121] LustreError: 367825:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9319.713636] LustreError: 367825:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9319.716503] LustreError: 367825:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9323.102391] LustreError: 367873:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9323.113815] LustreError: 367873:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9330.875209] Lustre: server umount lustre-OST0000 complete [ 9333.614906] LustreError: 368123:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9333.625173] LustreError: 368123:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9338.849543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9338.855545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.878282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.384136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.384939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.506906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9345.513112] Lustre: Skipped 2 previous similar messages [ 9346.598478] Lustre: server umount lustre-MDT0000 complete [ 9350.513842] LustreError: 365106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336171 with bad export cookie 13230564568306568929 [ 9350.528118] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.529300] LustreError: 365106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9360.021263] Lustre: server umount lustre-MDT0001 complete [ 9368.763936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9373.082324] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9376.031646] LustreError: 369266:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9381.956483] Lustre: 369605:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [ 9383.917656] Lustre: server umount MGS complete [ 9395.197503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.804650] LustreError: 370328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.819922] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9395.825374] Lustre: Skipped 1 previous similar message [ 9395.881156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.313325] LustreError: 370329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.334603] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9410.091022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.475581] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9415.290125] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9421.014898] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9425.504142] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9427.717695] Lustre: Permanently reactivating lustre-OST0000 [ 9427.757985] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9427.760798] Lustre: Skipped 2 previous similar messages [ 9454.048049] LustreError: 371836:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9463.328524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9463.928622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.938192] Lustre: Skipped 1 previous similar message [ 9471.814203] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9478.044806] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9483.744350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9483.756933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9483.780993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9484.061533] Lustre: server umount lustre-OST0000 complete [ 9489.888972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.909854] Lustre: Skipped 1 previous similar message [ 9495.012584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9495.028403] Lustre: Skipped 3 previous similar messages [ 9498.195530] Lustre: server umount lustre-MDT0000 complete [ 9499.117735] LustreError: 371041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9499.136114] LustreError: 371041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9501.890973] LustreError: 370309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336323 with bad export cookie 13230564568306569979 [ 9501.892402] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.919645] LustreError: 370309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9502.268185] Lustre: server umount lustre-MDT0001 complete [ 9513.427733] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9517.512406] Key type lgssc unregistered [ 9517.898703] LNet: 374340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9517.919948] LNetError: 374340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9517.962587] LNet: Removed LNI 192.168.204.146@tcp [ 9519.465542] Key type .llcrypt unregistered [ 9519.467761] Key type ._llcrypt unregistered [ 9541.722426] Key type ._llcrypt registered [ 9541.724985] Key type .llcrypt registered [ 9541.870069] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9557.526211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9558.319526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9558.438931] alg: No test for adler32 (adler32-zlib) [ 9559.691435] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 9559.998835] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9561.714602] Key type lgssc registered [ 9563.363770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9601.544991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9601.581085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.807485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9602.828303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9602.898941] Lustre: lustre-MDT0000: new disk, initializing [ 9602.978611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.005899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9607.095763] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9616.965449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.084572] Lustre: 377463:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9617.136389] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9617.141665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9617.150061] Lustre: Skipped 1 previous similar message [ 9617.234234] Lustre: lustre-MDT0001: new disk, initializing [ 9617.340839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9617.395977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9617.410861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9621.709270] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9629.085862] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9632.661977] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9638.725525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9639.024453] Lustre: lustre-OST0000: new disk, initializing [ 9639.030231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9639.041112] Lustre: 379205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9639.169402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.997455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9647.662199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9647.676089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9647.794743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9652.011807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9656.368330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9656.597624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9661.077746] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9661.400772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9668.584103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.597168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9668.623818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9668.630113] LustreError: Skipped 1 previous similar message [ 9668.637530] Lustre: Skipped 1 previous similar message [ 9669.735743] Lustre: server umount lustre-OST0000 complete [ 9673.698229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9673.708146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.720200] Lustre: Skipped 2 previous similar messages [ 9673.726713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9673.739794] Lustre: Skipped 1 previous similar message [ 9678.816920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9678.829346] Lustre: Skipped 1 previous similar message [ 9679.855038] Lustre: server umount lustre-MDT0000 complete [ 9684.529108] LustreError: 377454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336505 with bad export cookie 11369983483687679330 [ 9684.551895] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9684.576106] LustreError: 377454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9685.226438] Lustre: server umount lustre-MDT0001 complete [ 9697.080844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9699.878900] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:55:18 (1780336518) [ 9708.444393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.828959] LustreError: 381518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.854375] LustreError: 381518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9708.961479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9713.107801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9714.156529] LustreError: 381519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9719.268688] LustreError: 381518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.333863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.716753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9727.231662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9732.926065] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9737.105582] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9744.341582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9744.781251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.967175] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9757.746973] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9759.947690] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9762.533934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9770.472922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.502890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9770.505358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9770.516745] Lustre: Skipped 1 previous similar message [ 9775.070040] Lustre: server umount lustre-OST0000 complete [ 9780.710938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.712498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9780.726942] Lustre: Skipped 2 previous similar messages [ 9785.567831] Lustre: server umount lustre-MDT0000 complete [ 9785.824831] LustreError: 382245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.844962] LustreError: 382245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9789.766272] LustreError: 381498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336610 with bad export cookie 11369983483687680254 [ 9789.768438] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9789.778634] LustreError: 381498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9790.120842] Lustre: server umount lustre-MDT0001 complete [ 9803.115118] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9806.768260] Key type lgssc unregistered [ 9807.116770] LNet: 384891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9807.121748] LNetError: 384891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9807.150569] LNet: Removed LNI 192.168.204.146@tcp [ 9808.233483] Key type .llcrypt unregistered [ 9808.236202] Key type ._llcrypt unregistered [ 9836.464303] Key type ._llcrypt registered [ 9836.466051] Key type .llcrypt registered [ 9836.621677] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9836.973822] Key type .llcrypt unregistered [ 9836.977059] Key type ._llcrypt unregistered [ 9855.402835] Key type ._llcrypt registered [ 9855.406446] Key type .llcrypt registered [ 9855.581597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9856.789956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9856.922805] alg: No test for adler32 (adler32-zlib) [ 9858.064091] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 9858.492059] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9860.247227] Key type lgssc registered [ 9862.128866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9872.177546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9872.190607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9873.768254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.151233] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9883.106327] LustreError: 387072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.128808] LustreError: 387072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9887.535311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9887.792741] LustreError: 387071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.816378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9892.842200] LustreError: 387068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.858887] LustreError: 387068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9894.354517] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9897.956516] LustreError: 387071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.973490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9897.980179] LustreError: 387071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9900.652359] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9903.074657] LustreError: 387072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.084751] LustreError: 387072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9907.860245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.221751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.695354] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9923.446179] Lustre: server umount lustre-OST0000 complete [ 9927.275678] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9927.285439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.304547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.676351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.680668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9933.796575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9933.807972] Lustre: Skipped 2 previous similar messages [ 9936.924388] Lustre: server umount lustre-MDT0000 complete [ 9938.913695] LustreError: 387068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9938.940827] LustreError: 387068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9939.944460] Lustre: 386383:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336744/real 1780336744] req@000000003bb13fed x1866818313544704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780336760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9939.975421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.703672] LustreError: 387051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336761 with bad export cookie 12235716105097796830 [ 9940.710259] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9940.717961] LustreError: 387051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9941.167322] Lustre: server umount lustre-MDT0001 complete [ 9951.555722] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9954.284450] Key type lgssc unregistered [ 9954.547958] LNet: 390055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9954.560787] LNetError: 390055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9954.580320] LNet: Removed LNI 192.168.204.146@tcp [ 9955.303301] Key type .llcrypt unregistered [ 9955.307161] Key type ._llcrypt unregistered [ 9982.031874] Key type ._llcrypt registered [ 9982.035525] Key type .llcrypt registered [ 9982.230466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9983.287965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9983.348516] alg: No test for adler32 (adler32-zlib) [ 9984.642191] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 9985.043791] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9986.751175] Key type lgssc registered [ 9987.982501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9995.246461] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9995.281624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10010.850267] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10015.967988] LustreError: 391429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [10042.655880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10049.599859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10055.202133] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10060.961311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.482893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.593157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10065.608641] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10066.919429] LustreError: 391454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10066.946499] LustreError: 391454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10071.607034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.835318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.475372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10078.659389] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10082.031898] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10091.194472] Lustre: server umount lustre-OST0000 complete [10097.130685] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10097.140876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10097.153468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.753215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.766581] Lustre: Skipped 2 previous similar messages [10104.900163] Lustre: server umount lustre-MDT0000 complete [10107.871613] Lustre: 390767:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780336913/real 1780336913] req@000000008b2bab96 x1866818446716288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780336929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10107.875465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.896108] Lustre: 390767:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10107.942162] Lustre: Skipped 1 previous similar message [10107.975055] LustreError: 392365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.001809] LustreError: 392365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10109.863166] LustreError: 391435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336931 with bad export cookie 6708641644832857332 [10109.871431] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10109.880821] LustreError: 391435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10110.177535] Lustre: server umount lustre-MDT0001 complete [10121.715644] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10124.493282] Key type lgssc unregistered [10124.817499] LNet: 394653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10124.829262] LNetError: 394653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10124.852793] LNet: Removed LNI 192.168.204.146@tcp [10125.994451] Key type .llcrypt unregistered [10126.000760] Key type ._llcrypt unregistered [10149.018480] Key type ._llcrypt registered [10149.022585] Key type .llcrypt registered [10149.206845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10150.334698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10150.486068] alg: No test for adler32 (adler32-zlib) [10151.482699] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [10151.641812] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10153.311193] Key type lgssc registered [10154.583643] Lustre: Echo OBD driver; http://www.lustre.org/ [10161.445513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10161.471970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.999188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10167.502784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10173.410056] LustreError: 396056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.422194] LustreError: 396056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10176.813204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.298743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10182.041822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10188.212439] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10191.923964] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10198.323596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.735556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10205.208485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10207.979700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10210.633639] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10218.471693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10218.472149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10218.478429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10218.483571] Lustre: Skipped 1 previous similar message [10221.284567] Lustre: server umount lustre-MDT0000 complete [10223.585613] LustreError: 398295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.605590] LustreError: 398295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10223.722766] LustreError: 396035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.46@tcp arrived at 1780337044 with bad export cookie 13589628492199197973 [10223.741817] LustreError: 396035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10225.407037] LustreError: 396037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337046 with bad export cookie 13589628492199197077 [10225.414128] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10225.423470] LustreError: 396037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10227.681898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10227.689127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.708019] Lustre: Skipped 1 previous similar message [10227.713199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.719043] Lustre: Skipped 3 previous similar messages [10228.704646] LustreError: 396049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.728363] LustreError: 396049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10230.753697] LustreError: 396056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.757440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10230.776632] LustreError: 396056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10230.785408] Lustre: Skipped 1 previous similar message [10231.658830] Lustre: server umount lustre-OST0000 complete [10234.851691] LustreError: 396049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10234.866217] LustreError: 396049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10238.557692] Lustre: server umount lustre-MDT0001 complete [10248.242864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10250.927399] Key type lgssc unregistered [10251.168622] LNet: 399385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10251.173660] LNetError: 399385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10251.190433] LNet: Removed LNI 192.168.204.146@tcp [10251.879844] Key type .llcrypt unregistered [10251.882313] Key type ._llcrypt unregistered [10275.978400] Key type ._llcrypt registered [10275.980739] Key type .llcrypt registered [10276.129800] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10277.329023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10277.348937] alg: No test for adler32 (adler32-zlib) [10278.535561] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [10278.848797] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10280.559176] Key type lgssc registered [10282.131466] Lustre: Echo OBD driver; http://www.lustre.org/ [10291.860510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10291.903344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.642607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10298.997441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10302.948690] LustreError: 400785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.957434] LustreError: 400785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10308.071345] LustreError: 400786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.178489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.616513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.371507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10320.066826] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10324.214512] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10332.111489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10332.616938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10339.845610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10340.121054] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10346.202387] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10354.163305] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [10355.176537] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.180765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10355.190752] Lustre: Skipped 1 previous similar message [10355.215876] Lustre: Skipped 1 previous similar message [10357.135904] Lustre: server umount lustre-MDT0001 complete [10360.300328] LustreError: 400786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.322545] LustreError: 400786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10365.415233] LustreError: 400785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10365.415353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10365.419674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.425031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10369.483499] Lustre: server umount lustre-OST0000 complete [10370.537288] LustreError: 401497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.197960] Lustre: server umount lustre-MDT0000 complete [10385.594720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10388.049215] Key type lgssc unregistered [10388.352710] LNet: 404115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10388.362316] LNetError: 404115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10388.383905] LNet: Removed LNI 192.168.204.146@tcp [10389.036102] Key type .llcrypt unregistered [10389.038751] Key type ._llcrypt unregistered [10413.371763] Key type ._llcrypt registered [10413.379978] Key type .llcrypt registered [10413.522748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10428.609135] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10430.055825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10430.088558] alg: No test for adler32 (adler32-zlib) [10431.313278] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [10431.549833] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10433.263210] Key type lgssc registered [10434.466636] Lustre: Echo OBD driver; http://www.lustre.org/ [10470.172802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10485.788715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10485.895429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.223512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10487.356844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10487.513981] Lustre: lustre-MDT0000: new disk, initializing [10487.632729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10487.663361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10492.909811] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10508.597599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10508.819322] Lustre: lustre-OST0000: new disk, initializing [10508.823594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10508.826484] Lustre: Skipped 1 previous similar message [10508.835565] Lustre: 409083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10508.898500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10515.006381] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10517.557438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10517.573301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10517.621823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10522.454924] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10537.054492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.184445] Lustre: 408082:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10537.259927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10537.354247] Lustre: lustre-MDT0001: new disk, initializing [10537.480134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10537.518947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10537.527947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10541.882245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10551.128915] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10551.377957] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10556.428443] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10556.626369] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10563.571105] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [10566.113978] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.132236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10566.140691] Lustre: Skipped 1 previous similar message [10568.064354] Lustre: server umount lustre-MDT0001 complete [10576.353633] LustreError: 408090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10576.387288] LustreError: 408090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10581.473290] LustreError: 408089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.497835] LustreError: 408089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10586.598120] LustreError: 408090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.616163] LustreError: 408090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10589.153052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10589.171117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10589.200571] Lustre: Skipped 1 previous similar message [10589.209400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10591.712447] LustreError: 408094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.712614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10593.129884] Lustre: server umount lustre-OST0000 complete [10596.835396] LustreError: 408093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10603.472664] Lustre: server umount lustre-MDT0000 complete [10616.720515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10619.041620] Key type lgssc unregistered [10619.465873] LNet: 412658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10619.488827] LNetError: 412658:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10620.522691] LNet: Removed LNI 192.168.204.146@tcp [10621.468523] Key type .llcrypt unregistered [10621.470420] Key type ._llcrypt unregistered [10661.810176] Key type ._llcrypt registered [10661.813911] Key type .llcrypt registered [10661.910766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10662.235320] Key type .llcrypt unregistered [10662.237091] Key type ._llcrypt unregistered [10680.478202] Key type ._llcrypt registered [10680.480351] Key type .llcrypt registered [10680.582799] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10681.509358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10681.631695] alg: No test for adler32 (adler32-zlib) [10682.679548] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [10682.965142] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10684.655200] Key type lgssc registered [10685.809555] Lustre: Echo OBD driver; http://www.lustre.org/ [10712.039425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10712.057803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10713.324547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10713.358340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10713.452850] Lustre: lustre-MDT0000: new disk, initializing [10713.541678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10713.562766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10718.336654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10728.712265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.857160] Lustre: 416000:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10728.957803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10728.963174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10728.966580] Lustre: Skipped 1 previous similar message [10729.034833] Lustre: lustre-MDT0001: new disk, initializing [10729.118724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10729.158121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10729.172630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10734.166263] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10741.797622] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10745.339051] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10751.866701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10752.147818] Lustre: lustre-OST0000: new disk, initializing [10752.158340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10752.170108] Lustre: 417746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10752.273523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10758.258933] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10762.342750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10762.362321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10762.490682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10766.472536] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10775.520713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.536572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.559773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.083455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.090816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.119276] Lustre: server umount lustre-MDT0000 complete [10784.962291] LustreError: 415993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337606 with bad export cookie 3148719251078780308 [10784.964760] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10784.969954] LustreError: 415993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10785.223355] Lustre: server umount lustre-MDT0001 complete [10799.291534] Lustre: server umount lustre-OST0000 complete [10808.516842] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:13:48 (1780337628) [10814.615753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.971843] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.984817] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10815.039079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10819.704505] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10820.069434] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10820.096445] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10825.185859] LustreError: 419661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10829.373881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.650650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10834.464751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10839.716790] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10843.797218] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.667371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.880148] Lustre: Found index 0 for lustre-OST0000, updating log [10864.108932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10870.584093] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10876.251753] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10886.651752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10895.842090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10895.854577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.869162] Lustre: Skipped 1 previous similar message [10895.874824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10895.878917] Lustre: Skipped 2 previous similar messages [10898.726239] Lustre: server umount lustre-OST0000 complete [10916.835284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10916.854261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.874340] Lustre: Skipped 1 previous similar message [10916.881128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10916.891110] Lustre: Skipped 3 previous similar messages [10925.536908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10925.543603] Lustre: Skipped 7 previous similar messages [10929.119300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10929.416928] Lustre: server umount lustre-MDT0000 complete [10930.665431] LustreError: 419661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.684764] LustreError: 419661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10933.015422] LustreError: 419641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337754 with bad export cookie 3148719251078781743 [10933.020394] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10933.030181] LustreError: 419641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10933.409705] Lustre: server umount lustre-MDT0001 complete [10933.856333] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [10941.616614] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:16:01 (1780337761) [11563.800262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.465888] LustreError: 425763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11564.611727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11567.943631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11569.637912] LustreError: 425764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.761927] LustreError: 425763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.810303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.185633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11580.048343] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11586.366625] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11590.718783] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11610.481098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11610.767865] Lustre: Found index 0 for lustre-OST0000, updating log [11611.175247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11612.214281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11617.424461] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11623.071363] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [11623.473859] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11626.791215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11626.805015] Lustre: Skipped 1 previous similar message [11626.811150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11640.287153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11640.458273] Lustre: server umount lustre-OST0000 complete [11645.346195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.358744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.463354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.475715] Lustre: Skipped 8 previous similar messages [11650.681499] Lustre: server umount lustre-MDT0000 complete [11651.559248] LustreError: 425763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.582179] LustreError: 425763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11654.788701] LustreError: 427387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338475 with bad export cookie 3148719251078782828 [11654.801994] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11654.808560] LustreError: 427387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11655.285739] Lustre: server umount lustre-MDT0001 complete [11686.000684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11696.481446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11733.854517] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:29:13 (1780338553) [11755.530938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11758.350860] Key type lgssc unregistered [11758.636573] LNet: 431986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11758.647200] LNetError: 431986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11871.337189] LNet: Removed LNI 192.168.204.146@tcp [11872.400948] Key type .llcrypt unregistered [11872.410779] Key type ._llcrypt unregistered [11891.994149] Key type ._llcrypt registered [11891.996616] Key type .llcrypt registered [11892.165842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11893.132168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11893.272096] alg: No test for adler32 (adler32-zlib) [11894.514994] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [11894.788105] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11896.543857] Key type lgssc registered [11897.737571] Lustre: Echo OBD driver; http://www.lustre.org/ [11902.222872] LNet: Added LNI 192.168.204.146@tcp100 [8/256/0/180] [11903.242764] LNet: Added LNI 192.168.204.146@tcp101 [8/256/0/180] [11904.238935] LNet: Added LNI 192.168.204.146@tcp102 [8/256/0/180] [11905.160528] LNet: Added LNI 192.168.204.146@tcp103 [8/256/0/180] [11906.114265] LNet: Added LNI 192.168.204.146@tcp104 [8/256/0/180] [11907.070638] LNet: Added LNI 192.168.204.146@tcp105 [8/256/0/180] [11908.147385] LNet: Added LNI 192.168.204.146@tcp106 [8/256/0/180] [11909.004560] LNet: Added LNI 192.168.204.146@tcp107 [8/256/0/180] [11910.021777] LNet: Added LNI 192.168.204.146@tcp108 [8/256/0/180] [11911.067318] LNet: Added LNI 192.168.204.146@tcp109 [8/256/0/180] [11911.990689] LNet: Added LNI 192.168.204.146@tcp110 [8/256/0/180] [11912.893565] LNet: Added LNI 192.168.204.146@tcp111 [8/256/0/180] [11913.616964] LNet: Added LNI 192.168.204.146@tcp112 [8/256/0/180] [11914.363473] LNet: Added LNI 192.168.204.146@tcp113 [8/256/0/180] [11915.291312] LNet: Added LNI 192.168.204.146@tcp114 [8/256/0/180] [11916.190845] LNet: Added LNI 192.168.204.146@tcp115 [8/256/0/180] [11917.037708] LNet: Added LNI 192.168.204.146@tcp116 [8/256/0/180] [11917.927770] LNet: Added LNI 192.168.204.146@tcp117 [8/256/0/180] [11918.839918] LNet: Added LNI 192.168.204.146@tcp118 [8/256/0/180] [11919.665527] LNet: Added LNI 192.168.204.146@tcp119 [8/256/0/180] [11920.572665] LNet: Added LNI 192.168.204.146@tcp120 [8/256/0/180] [11921.332163] LNet: Added LNI 192.168.204.146@tcp121 [8/256/0/180] [11922.155920] LNet: Added LNI 192.168.204.146@tcp122 [8/256/0/180] [11922.951936] LNet: Added LNI 192.168.204.146@tcp123 [8/256/0/180] [11923.771261] LNet: Added LNI 192.168.204.146@tcp124 [8/256/0/180] [11924.598437] LNet: Added LNI 192.168.204.146@tcp125 [8/256/0/180] [11925.505542] LNet: Added LNI 192.168.204.146@tcp126 [8/256/0/180] [11926.444301] LNet: Added LNI 192.168.204.146@tcp127 [8/256/0/180] [11927.446623] LNet: Added LNI 192.168.204.146@tcp128 [8/256/0/180] [11928.260381] LNet: Added LNI 192.168.204.146@tcp129 [8/256/0/180] [11929.050462] LNet: Added LNI 192.168.204.146@tcp130 [8/256/0/180] [11929.899690] LNet: Added LNI 192.168.204.146@tcp131 [8/256/0/180] [11930.817085] LNet: Added LNI 192.168.204.146@tcp132 [8/256/0/180] [11931.615263] LNet: Added LNI 192.168.204.146@tcp133 [8/256/0/180] [11932.502588] LNet: Added LNI 192.168.204.146@tcp134 [8/256/0/180] [11933.361906] LNet: Added LNI 192.168.204.146@tcp135 [8/256/0/180] [11939.951636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11939.978568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.183890] Lustre: 435003:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [11941.281292] LNetError: 435026:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11941.322922] LNetError: Protocol error connecting to 192.168.204.146@tcp115 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11941.336168] LNetError: Skipped 2 previous similar messages [11941.390388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11941.422057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11941.503768] Lustre: lustre-MDT0000: new disk, initializing [11941.639790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11941.661238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11941.920458] LNet: 7 local NIs in recovery (showing 7): 192.168.204.146@tcp126, 192.168.204.146@tcp128, 192.168.204.146@tcp127, 192.168.204.146@tcp129, 192.168.204.146@tcp130, 192.168.204.146@tcp131, 192.168.204.146@tcp132 [11942.371666] LNetError: 432459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11942.392456] LNetError: 432459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11942.407169] LNetError: Protocol error connecting to 192.168.204.146@tcp101 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11942.416498] LNetError: Skipped 5 previous similar messages [11945.062837] LNetError: 435028:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11945.096762] LNetError: 435028:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [11945.108870] LNetError: Protocol error connecting to 192.168.204.146@tcp126 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11945.138361] LNetError: Skipped 4 previous similar messages [11946.488957] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11949.153423] LNetError: 435038:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11949.173590] LNetError: Protocol error connecting to 192.168.204.146@tcp126 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11957.349144] LNetError: 432461:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11957.363697] LNetError: Protocol error connecting to 192.168.204.146@tcp126 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11957.767073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.946755] Lustre: 435022:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11958.021668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11958.027567] Lustre: Skipped 1 previous similar message [11958.100683] Lustre: lustre-MDT0001: new disk, initializing [11958.155574] Lustre: 435807:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [11958.167588] Lustre: 435807:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11958.280735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11958.305508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11958.317308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11963.249878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11964.480691] Lustre: 436108:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [11964.495882] Lustre: 436108:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11965.538891] LNetError: 435038:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11965.559048] LNetError: 435038:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [11965.570796] LNetError: Protocol error connecting to 192.168.204.146@tcp133 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11965.580627] LNetError: Skipped 9 previous similar messages [11971.337311] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11975.739622] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11982.944543] LNetError: 435025:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [11982.960778] LNetError: 435025:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11982.976460] LNetError: Protocol error connecting to 192.168.204.146@tcp101 at host 192.168.204.146:988: is it running a compatible version of Lustre? [11982.985071] LNetError: Skipped 5 previous similar messages [11983.139831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11983.405664] Lustre: lustre-OST0000: new disk, initializing [11983.409118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11983.414095] Lustre: 436773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11983.499946] Lustre: 436778:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [11983.506461] Lustre: 436778:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11983.551108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11990.897747] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11993.172864] Lustre: 437198:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [11993.195583] Lustre: 437198:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11993.266142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11993.282622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11993.408492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11999.675956] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12002.399349] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp133 [12008.931044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12008.933060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.939630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12008.943413] LustreError: Skipped 1 previous similar message [12008.971330] Lustre: Skipped 1 previous similar message [12013.763701] Lustre: server umount lustre-OST0000 complete [12015.780028] LNetError: 432460:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [12015.799838] LNetError: 432460:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [12015.805674] LNetError: Protocol error connecting to 192.168.204.146@tcp121 at host 192.168.204.146:988: is it running a compatible version of Lustre? [12015.814425] LNetError: Skipped 12 previous similar messages [12019.169930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.178965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.187583] Lustre: Skipped 1 previous similar message [12019.206672] Lustre: Skipped 2 previous similar messages [12024.441215] Lustre: server umount lustre-MDT0000 complete [12028.703921] LustreError: 435013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338849 with bad export cookie 5171574468189927410 [12028.707037] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.732617] LustreError: 435013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12029.047695] Lustre: server umount lustre-MDT0001 complete [12041.715923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12044.943073] Key type lgssc unregistered [12045.525436] LNet: 438557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12045.530889] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12045.541126] LNet: Removed LNI 192.168.204.146@tcp [12046.562190] LNet: Removed LNI 192.168.204.146@tcp100 [12046.564250] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12046.576671] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12046.585973] LNet: Removed LNI 192.168.204.146@tcp101 [12047.647185] LNet: Removed LNI 192.168.204.146@tcp102 [12047.651210] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12047.668900] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12047.699812] LNet: Removed LNI 192.168.204.146@tcp103 [12048.735188] LNet: Removed LNI 192.168.204.146@tcp104 [12049.760195] LNet: Removed LNI 192.168.204.146@tcp105 [12049.774853] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12049.786735] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12049.792278] LNet: Removed LNI 192.168.204.146@tcp106 [12050.849480] LNet: Removed LNI 192.168.204.146@tcp107 [12051.871198] LNet: Removed LNI 192.168.204.146@tcp108 [12052.895325] LNet: Removed LNI 192.168.204.146@tcp109 [12053.919292] LNet: Removed LNI 192.168.204.146@tcp110 [12053.932308] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12053.941734] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12053.968725] LNet: Removed LNI 192.168.204.146@tcp111 [12055.014174] LNet: Removed LNI 192.168.204.146@tcp112 [12056.031415] LNet: Removed LNI 192.168.204.146@tcp113 [12057.055170] LNet: Removed LNI 192.168.204.146@tcp114 [12058.080410] LNet: Removed LNI 192.168.204.146@tcp115 [12059.103967] LNet: Removed LNI 192.168.204.146@tcp116 [12060.127336] LNet: Removed LNI 192.168.204.146@tcp117 [12061.151506] LNet: Removed LNI 192.168.204.146@tcp118 [12062.177545] LNet: Removed LNI 192.168.204.146@tcp119 [12062.184170] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12062.210270] LNetError: 438557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12062.221720] LNet: Removed LNI 192.168.204.146@tcp120 [12063.263743] LNet: Removed LNI 192.168.204.146@tcp121 [12064.287160] LNet: Removed LNI 192.168.204.146@tcp122 [12065.312568] LNet: Removed LNI 192.168.204.146@tcp123 [12066.335166] LNet: Removed LNI 192.168.204.146@tcp124 [12067.360049] LNet: Removed LNI 192.168.204.146@tcp125 [12068.384306] LNet: Removed LNI 192.168.204.146@tcp126 [12069.407349] LNet: Removed LNI 192.168.204.146@tcp127 [12070.431234] LNet: Removed LNI 192.168.204.146@tcp128 [12071.457098] LNet: Removed LNI 192.168.204.146@tcp129 [12072.479155] LNet: Removed LNI 192.168.204.146@tcp130 [12073.503245] LNet: Removed LNI 192.168.204.146@tcp131 [12074.527183] LNet: Removed LNI 192.168.204.146@tcp132 [12074.529964] LNet: Removed LNI 192.168.204.146@tcp133 [12075.551162] LNet: Removed LNI 192.168.204.146@tcp134 [12076.647412] LNet: Removed LNI 192.168.204.146@tcp135 [12077.570906] Key type .llcrypt unregistered [12077.573914] Key type ._llcrypt unregistered [12101.333943] Key type ._llcrypt registered [12101.335637] Key type .llcrypt registered [12101.425984] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12118.563775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12119.387274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12119.451500] alg: No test for adler32 (adler32-zlib) [12120.509897] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [12120.707849] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12122.439405] Key type lgssc registered [12123.574232] Lustre: Echo OBD driver; http://www.lustre.org/ [12159.529703] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:36:19 (1780338979) [12177.462190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12180.277545] Key type lgssc unregistered [12180.576411] LNet: 442596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12180.589204] LNetError: 442596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12180.604884] LNet: Removed LNI 192.168.204.146@tcp [12181.594768] Key type .llcrypt unregistered [12181.596324] Key type ._llcrypt unregistered [12198.364471] Key type ._llcrypt registered [12198.367704] Key type .llcrypt registered [12198.471887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12199.565358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12199.638981] alg: No test for adler32 (adler32-zlib) [12200.762757] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [12201.112181] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12202.855169] Key type lgssc registered [12203.964297] Lustre: Echo OBD driver; http://www.lustre.org/ [12213.700082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12213.714267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.023733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12215.072642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12215.192658] Lustre: lustre-MDT0000: new disk, initializing [12215.321552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12215.350052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12220.557309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12231.725236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.868667] Lustre: 443792:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12231.965285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12231.971557] Lustre: Skipped 1 previous similar message [12232.114634] Lustre: lustre-MDT0001: new disk, initializing [12232.290705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12232.354621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12232.371059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12237.542768] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12246.144996] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12250.126705] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12257.602186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12257.921064] Lustre: lustre-OST0000: new disk, initializing [12257.932406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12257.943685] Lustre: 445539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12258.057943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12264.445284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12265.064443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12265.081239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12265.192964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12272.485229] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12276.801229] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lnet_if_list [12283.577547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12283.610751] LNet: Added LNI 192.168.204.146@tcp10 [8/256/0/180] [12290.795923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12290.836794] LNet: Added LNI 192.168.204.146@tcp11 [8/256/0/180] [12296.389121] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12296.428879] LNet: Added LNI 192.168.204.146@tcp12 [8/256/0/180] [12301.772955] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12301.828822] LNet: Added LNI 192.168.204.146@tcp13 [8/256/0/180] [12306.781878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12306.832466] LNet: Added LNI 192.168.204.146@tcp14 [8/256/0/180] [12311.997455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12312.028679] LNet: Added LNI 192.168.204.146@tcp15 [8/256/0/180] [12317.168544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12317.193496] LNet: Added LNI 192.168.204.146@tcp16 [8/256/0/180] [12322.211029] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12322.238492] LNet: Added LNI 192.168.204.146@tcp17 [8/256/0/180] [12327.257314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12327.319315] LNet: Added LNI 192.168.204.146@tcp18 [8/256/0/180] [12332.870509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12332.898915] LNet: Added LNI 192.168.204.146@tcp19 [8/256/0/180] [12338.193328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12338.218580] LNet: Added LNI 192.168.204.146@tcp20 [8/256/0/180] [12343.477211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12343.517736] LNet: Added LNI 192.168.204.146@tcp21 [8/256/0/180] [12349.006920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12349.033267] LNet: Added LNI 192.168.204.146@tcp22 [8/256/0/180] [12354.840729] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12354.874065] LNet: Added LNI 192.168.204.146@tcp23 [8/256/0/180] [12361.453877] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12361.502322] LNet: Added LNI 192.168.204.146@tcp24 [8/256/0/180] [12368.183824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12368.214761] LNet: Added LNI 192.168.204.146@tcp25 [8/256/0/180] [12374.780914] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12374.821221] LNet: Added LNI 192.168.204.146@tcp26 [8/256/0/180] [12381.239193] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12381.288927] LNet: Added LNI 192.168.204.146@tcp27 [8/256/0/180] [12388.391963] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12388.422506] LNet: Added LNI 192.168.204.146@tcp28 [8/256/0/180] [12394.170498] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12394.191467] LNet: Added LNI 192.168.204.146@tcp29 [8/256/0/180] [12400.425938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12400.458529] LNet: Added LNI 192.168.204.146@tcp30 [8/256/0/180] [12407.786248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12407.804485] LNet: Added LNI 192.168.204.146@tcp31 [8/256/0/180] [12415.567676] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12415.617375] LNet: Added LNI 192.168.204.146@tcp32 [8/256/0/180] [12424.735460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12424.823887] LNet: Added LNI 192.168.204.146@tcp33 [8/256/0/180] [12432.635636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12432.667531] LNet: Added LNI 192.168.204.146@tcp34 [8/256/0/180] [12440.505591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12440.559374] LNet: Added LNI 192.168.204.146@tcp35 [8/256/0/180] [12447.765352] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12447.793895] LNet: Added LNI 192.168.204.146@tcp36 [8/256/0/180] [12455.006114] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12455.069860] LNet: Added LNI 192.168.204.146@tcp37 [8/256/0/180] [12463.174232] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12463.214468] LNet: Added LNI 192.168.204.146@tcp38 [8/256/0/180] [12471.848787] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12471.904814] LNet: Added LNI 192.168.204.146@tcp39 [8/256/0/180] [12479.848377] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12479.894210] LNet: Added LNI 192.168.204.146@tcp40 [8/256/0/180] [12487.488392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12487.509786] LNet: Added LNI 192.168.204.146@tcp41 [8/256/0/180] [12495.574540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12495.609775] LNet: Added LNI 192.168.204.146@tcp42 [8/256/0/180] [12504.750931] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12504.808317] LNet: Added LNI 192.168.204.146@tcp43 [8/256/0/180] [12514.026527] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12514.064835] LNet: Added LNI 192.168.204.146@tcp44 [8/256/0/180] [12522.612207] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12522.656706] LNet: Added LNI 192.168.204.146@tcp45 [8/256/0/180] [12531.957590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12531.986638] LNet: Added LNI 192.168.204.146@tcp46 [8/256/0/180] [12541.690571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12541.738423] LNet: Added LNI 192.168.204.146@tcp47 [8/256/0/180] [12551.033845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12551.074629] LNet: Added LNI 192.168.204.146@tcp48 [8/256/0/180] [12560.975860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12561.023736] LNet: Added LNI 192.168.204.146@tcp49 [8/256/0/180] [12570.869651] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12570.902613] LNet: Added LNI 192.168.204.146@tcp50 [8/256/0/180] [12593.127665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.130893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12593.145300] Lustre: Skipped 1 previous similar message [12593.168295] Lustre: Skipped 2 previous similar messages [12597.899255] Lustre: server umount lustre-MDT0000 complete [12601.703159] LustreError: 444573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339422 with bad export cookie 12122582086661133459 [12601.704663] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12601.711922] LustreError: 444573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12602.036368] Lustre: server umount lustre-MDT0001 complete [12619.743111] Lustre: 443066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339424/real 1780339424] req@000000000dd8ad9f x1866820770510336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12619.782223] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.792240] Lustre: Skipped 1 previous similar message [12621.672726] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12622.816726] Lustre: 443064:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339427/real 1780339427] req@00000000ab001af8 x1866820770510464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12623.843790] Lustre: 443063:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339429/real 1780339429] req@000000007f6fa72d x1866820770510592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12626.913395] Lustre: 443066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339432/real 1780339432] req@00000000617c6b24 x1866820770510848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12631.733304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12632.095503] Lustre: 443066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339437/real 1780339437] req@0000000091d262f6 x1866820770510976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12646.739870] LustreError: 445533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12646.767819] LustreError: 445533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12646.834671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12647.851784] LustreError: 445534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12647.859577] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12649.893204] LustreError: 445533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.767607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12655.075048] LustreError: 445534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12660.192259] LustreError: 445533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12660.384494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12660.962320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12664.039618] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12665.940582] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12672.800511] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12677.595405] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12700.131099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.136427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12700.150170] Lustre: Skipped 1 previous similar message [12700.169073] Lustre: Skipped 1 previous similar message [12705.257526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12705.288533] Lustre: Skipped 1 previous similar message [12705.982100] Lustre: server umount lustre-OST0000 complete [12715.488866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12715.496482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12715.527487] Lustre: Skipped 1 previous similar message [12715.539237] Lustre: Skipped 1 previous similar message [12718.957869] Lustre: server umount lustre-MDT0000 complete [12720.615929] LustreError: 468955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.648332] LustreError: 468955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12724.046365] LustreError: 443784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339545 with bad export cookie 12122582086661136140 [12724.048625] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12724.051293] LustreError: 443784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12724.387260] Lustre: server umount lustre-MDT0001 complete [12737.383581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12741.098682] Key type lgssc unregistered [12741.621645] LNet: 471517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12741.628886] LNetError: 471517:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12741.634971] LNet: Removed LNI 192.168.204.146@tcp [12741.638388] LNet: Removed LNI 192.168.204.146@tcp10 [12741.642355] LNet: Removed LNI 192.168.204.146@tcp11 [12741.646379] LNet: Removed LNI 192.168.204.146@tcp12 [12741.649169] LNet: Removed LNI 192.168.204.146@tcp13 [12741.652370] LNet: Removed LNI 192.168.204.146@tcp14 [12741.655587] LNet: Removed LNI 192.168.204.146@tcp15 [12741.658841] LNet: Removed LNI 192.168.204.146@tcp16 [12741.660996] LNet: Removed LNI 192.168.204.146@tcp17 [12741.664775] LNet: Removed LNI 192.168.204.146@tcp18 [12741.672410] LNet: Removed LNI 192.168.204.146@tcp19 [12741.674483] LNet: Removed LNI 192.168.204.146@tcp20 [12741.682662] LNet: Removed LNI 192.168.204.146@tcp21 [12741.685148] LNet: Removed LNI 192.168.204.146@tcp22 [12741.691591] LNet: Removed LNI 192.168.204.146@tcp23 [12741.694976] LNet: Removed LNI 192.168.204.146@tcp24 [12741.699485] LNet: Removed LNI 192.168.204.146@tcp25 [12741.702686] LNet: Removed LNI 192.168.204.146@tcp26 [12741.705531] LNet: Removed LNI 192.168.204.146@tcp27 [12741.708452] LNet: Removed LNI 192.168.204.146@tcp28 [12741.711574] LNet: Removed LNI 192.168.204.146@tcp29 [12741.714286] LNet: Removed LNI 192.168.204.146@tcp30 [12741.716545] LNet: Removed LNI 192.168.204.146@tcp31 [12741.719236] LNet: Removed LNI 192.168.204.146@tcp32 [12741.722211] LNet: Removed LNI 192.168.204.146@tcp33 [12741.725434] LNet: Removed LNI 192.168.204.146@tcp34 [12741.728295] LNet: Removed LNI 192.168.204.146@tcp35 [12741.743249] LNet: Removed LNI 192.168.204.146@tcp36 [12741.749863] LNet: Removed LNI 192.168.204.146@tcp37 [12741.753581] LNet: Removed LNI 192.168.204.146@tcp38 [12741.759848] LNet: Removed LNI 192.168.204.146@tcp39 [12741.766213] LNet: Removed LNI 192.168.204.146@tcp40 [12741.770085] LNet: Removed LNI 192.168.204.146@tcp41 [12741.775364] LNet: Removed LNI 192.168.204.146@tcp42 [12741.779189] LNet: Removed LNI 192.168.204.146@tcp43 [12741.781752] LNet: Removed LNI 192.168.204.146@tcp44 [12741.786318] LNet: Removed LNI 192.168.204.146@tcp45 [12741.795078] LNet: Removed LNI 192.168.204.146@tcp46 [12741.801271] LNet: Removed LNI 192.168.204.146@tcp47 [12741.807442] LNet: Removed LNI 192.168.204.146@tcp48 [12741.814318] LNet: Removed LNI 192.168.204.146@tcp49 [12741.839518] LNet: Removed LNI 192.168.204.146@tcp50 [12742.590583] Key type .llcrypt unregistered [12742.592337] Key type ._llcrypt unregistered [12765.660448] Key type ._llcrypt registered [12765.662577] Key type .llcrypt registered [12765.812454] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12780.817206] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12782.138535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12782.185734] alg: No test for adler32 (adler32-zlib) [12783.314370] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [12783.599032] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12785.287200] Key type lgssc registered [12786.841702] Lustre: Echo OBD driver; http://www.lustre.org/ [12823.804150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12823.826331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12825.106641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12825.147119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12825.267204] Lustre: lustre-MDT0000: new disk, initializing [12825.375559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12825.406746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12830.382199] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12841.625869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.794348] Lustre: 474682:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12841.888847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12841.894354] Lustre: Skipped 1 previous similar message [12842.030392] Lustre: lustre-MDT0001: new disk, initializing [12842.105790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12842.172288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12842.197866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12847.238823] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12855.117794] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12859.930837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12867.642688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12867.939230] Lustre: *** cfs_fail_loc=90f, val=0*** [12868.041588] Lustre: lustre-OST0000: new disk, initializing [12868.045491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12868.059435] Lustre: 476473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12868.159958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12875.188338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12876.911668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12876.922717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12877.092699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12883.870752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12892.644520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12892.647360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12892.658096] LustreError: Skipped 1 previous similar message [12892.670515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12892.684480] Lustre: Skipped 1 previous similar message [12896.795634] Lustre: server umount lustre-OST0000 complete [12902.883878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12902.899166] Lustre: Skipped 1 previous similar message [12902.903530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12902.916331] Lustre: Skipped 1 previous similar message [12906.760747] Lustre: server umount lustre-MDT0000 complete [12910.586119] LustreError: 474675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339731 with bad export cookie 4381563941717659524 [12910.604335] LustreError: 474675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12910.604746] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12910.898579] Lustre: server umount lustre-MDT0001 complete [12923.113710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12925.641303] Key type lgssc unregistered [12925.975747] LNet: 478249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12925.983317] LNetError: 478249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12925.997732] LNet: Removed LNI 192.168.204.146@tcp [12927.004947] Key type .llcrypt unregistered [12927.006401] Key type ._llcrypt unregistered [12950.136807] Key type ._llcrypt registered [12950.142085] Key type .llcrypt registered [12950.326760] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12951.428679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12951.516069] alg: No test for adler32 (adler32-zlib) [12952.544966] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [12952.878446] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12954.560051] Key type lgssc registered [12955.896491] Lustre: Echo OBD driver; http://www.lustre.org/ [12964.618982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12964.664529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12966.178848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12970.086597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12976.610575] LustreError: 479505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12976.625800] LustreError: 479505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12977.796558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.074119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12982.049377] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12987.261634] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12990.990323] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12997.669840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12998.077696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13003.227881] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13008.341384] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13021.406158] loop0: detected capacity change from 0 to 204800000 [13021.611639] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13021.618955] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13021.629183] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13027.160278] loop0: detected capacity change from 0 to 204800000 [13027.261780] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13027.272165] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13027.288691] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13032.248431] loop0: detected capacity change from 0 to 204800000 [13037.813830] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13038.114025] Lustre: fs15246-OST0000: new disk, initializing [13038.122671] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13038.147157] Lustre: 482762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13038.258584] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13045.123386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13054.280867] loop1: detected capacity change from 0 to 204800000 [13059.705852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13059.910139] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13059.997139] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13060.122266] Lustre: fs15246-MDT0000: new disk, initializing [13060.466972] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13060.533430] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13060.683543] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13061.729581] Lustre: 483866:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000004d0f1c7e x1866821558909056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780339898 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13066.773209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13066.880379] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13066.894195] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13066.940691] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13092.335070] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13092.349631] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13094.131708] Lustre: server umount fs15246-MDT0000 complete [13097.460959] LustreError: 484273:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.565124] LustreError: 479502:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13103.756670] Lustre: server umount fs15246-OST0000 complete [13117.927723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13117.933583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13117.940849] Lustre: Skipped 1 previous similar message [13117.950386] Lustre: Skipped 1 previous similar message [13122.297151] Lustre: server umount lustre-OST0000 complete [13126.627293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13126.646733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.680350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13132.590251] Lustre: server umount lustre-MDT0000 complete [13133.290431] LustreError: 480229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13133.318879] LustreError: 480229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13137.159206] LustreError: 479484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339958 with bad export cookie 7276324970119155670 [13137.173509] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13137.180823] LustreError: 479484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13137.614514] Lustre: server umount lustre-MDT0001 complete [13151.506583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13154.793376] Key type lgssc unregistered [13155.199525] LNet: 486931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13155.211112] LNetError: 486931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13155.247868] LNet: Removed LNI 192.168.204.146@tcp [13156.248400] Key type .llcrypt unregistered [13156.251562] Key type ._llcrypt unregistered [13176.005694] Key type ._llcrypt registered [13176.009772] Key type .llcrypt registered [13176.132236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13177.191795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13177.269169] alg: No test for adler32 (adler32-zlib) [13178.473946] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [13178.742984] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13180.463728] Key type lgssc registered [13181.524377] Lustre: Echo OBD driver; http://www.lustre.org/ [13221.228138] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13231.920854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13270.636440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13270.670214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13272.020613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13272.060464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13272.217339] Lustre: lustre-MDT0000: new disk, initializing [13272.406878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13272.464783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13277.595388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13288.630589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13288.743124] Lustre: 491349:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13288.797429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13288.802973] Lustre: Skipped 1 previous similar message [13288.901097] Lustre: lustre-MDT0001: new disk, initializing [13289.045088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13289.074477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13289.082794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13293.397920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13300.990160] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13304.893241] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13310.866449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13311.101929] Lustre: lustre-OST0000: new disk, initializing [13311.105755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13311.112329] Lustre: 493094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13311.220965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13317.836235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13317.841641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13317.894654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13318.043508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13325.574340] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13331.025889] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13331.274501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13336.971656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13337.370589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13343.712309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13343.719773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.729021] LustreError: Skipped 1 previous similar message [13343.730880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13343.752060] Lustre: Skipped 1 previous similar message [13345.726404] Lustre: server umount lustre-OST0000 complete [13350.369815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13350.380356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13350.397291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13350.405346] Lustre: Skipped 1 previous similar message [13353.952667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13353.953161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13355.921303] Lustre: server umount lustre-MDT0000 complete [13359.873828] LustreError: 491341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340181 with bad export cookie 12823540279009664680 [13359.880789] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13359.889721] LustreError: 491341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13360.291460] Lustre: server umount lustre-MDT0001 complete [13388.003102] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13401.848810] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13437.030802] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13251 sec ========= 14:57:36 (1780340256) [13438.993695] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:57:38 (1780340258) === [13443.576853] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:57:42 (1780340262) === [13486.580558] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13489.650647] Key type lgssc unregistered [13489.937223] LNet: 499561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13489.941582] LNetError: 499561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13489.956563] LNet: Removed LNI 192.168.204.146@tcp [13490.775205] Key type .llcrypt unregistered [13490.780150] Key type ._llcrypt unregistered