[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471843454 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009121] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011028] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.020562] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025390] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.030420] *** VALIDATE proc *** [ 0.031250] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.034017] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042090] debug: unmapping init [mem 0xffffffff88659000-0xffffffff88660fff] [ 0.044172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045580] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053288] rcu: Hierarchical SRCU implementation. [ 0.055332] smp: Bringing up secondary CPUs ... [ 0.056549] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.062016] smp: Brought up 1 node, 4 CPUs [ 0.064030] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235748] node 0 deferred pages initialised in 169ms [ 0.240151] devtmpfs: initialized [ 0.241276] x86/mm: Memory block size: 128MB [ 0.244804] gcov: version magic: 0x41383552 [ 0.247218] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252289] pinctrl core: initialized pinctrl subsystem [ 0.254152] [ 0.254442] ************************************************************* [ 0.256013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259014] ** ** [ 0.262015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263009] ** ** [ 0.266011] ** This means that this kernel is built to expose internal ** [ 0.269013] ** IOMMU data structures, which may compromise security on ** [ 0.271009] ** your system. ** [ 0.273009] ** ** [ 0.276014] ** If you see this message and you are not debugging the ** [ 0.278010] ** kernel, report this immediately to your vendor! ** [ 0.280010] ** ** [ 0.283012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286019] ************************************************************* [ 0.289752] NET: Registered protocol family 16 [ 0.291444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299024] cpuidle: using governor menu [ 0.300784] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.303576] PCI: Using configuration type 1 for base access [ 0.306127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.324161] cryptd: max_cpu_qlen set to 1000 [ 0.326308] ACPI: Added _OSI(Module Device) [ 0.327020] ACPI: Added _OSI(Processor Device) [ 0.330026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.331018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.336939] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343324] ACPI: Interpreter enabled [ 0.345059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.346012] ACPI: Using IOAPIC for interrupt routing [ 0.348096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.351364] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.360804] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.364034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.367021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.370093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376497] acpiphp: Slot [2] registered [ 0.377113] acpiphp: Slot [5] registered [ 0.379101] acpiphp: Slot [6] registered [ 0.380096] acpiphp: Slot [7] registered [ 0.382110] acpiphp: Slot [8] registered [ 0.383104] acpiphp: Slot [9] registered [ 0.384090] acpiphp: Slot [10] registered [ 0.386088] acpiphp: Slot [3] registered [ 0.387096] acpiphp: Slot [4] registered [ 0.389089] acpiphp: Slot [11] registered [ 0.391126] acpiphp: Slot [12] registered [ 0.393088] acpiphp: Slot [13] registered [ 0.394065] acpiphp: Slot [14] registered [ 0.396122] acpiphp: Slot [15] registered [ 0.397101] acpiphp: Slot [16] registered [ 0.399096] acpiphp: Slot [17] registered [ 0.400127] acpiphp: Slot [18] registered [ 0.402064] acpiphp: Slot [19] registered [ 0.403085] acpiphp: Slot [20] registered [ 0.404055] acpiphp: Slot [21] registered [ 0.405073] acpiphp: Slot [22] registered [ 0.407070] acpiphp: Slot [23] registered [ 0.408051] acpiphp: Slot [24] registered [ 0.409077] acpiphp: Slot [25] registered [ 0.411124] acpiphp: Slot [26] registered [ 0.412095] acpiphp: Slot [27] registered [ 0.414106] acpiphp: Slot [28] registered [ 0.416101] acpiphp: Slot [29] registered [ 0.417092] acpiphp: Slot [30] registered [ 0.419104] acpiphp: Slot [31] registered [ 0.420041] PCI host bridge to bus 0000:00 [ 0.422020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.433024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.438780] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.441190] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451751] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.456111] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.459022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.464016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.467334] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.470674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.473055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.476936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.482016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.498023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.503015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508768] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.515016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.538014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548427] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.556015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.579016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596112] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.627017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.636992] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.661016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.680017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.693404] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.705016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.712015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.733016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.741584] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.751015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.774014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.785163] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.787338] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.801113] iommu: Default domain type: Passthrough [ 0.803386] SCSI subsystem initialized [ 0.805146] ACPI: bus type USB registered [ 0.807096] usbcore: registered new interface driver usbfs [ 0.809093] usbcore: registered new interface driver hub [ 0.812081] usbcore: registered new device driver usb [ 0.814162] pps_core: LinuxPPS API ver. 1 registered [ 0.816011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.820059] PTP clock support registered [ 0.823087] EDAC MC: Ver: 3.0.0 [ 0.825205] PCI: Using ACPI for IRQ routing [ 0.828330] NetLabel: Initializing [ 0.830018] NetLabel: domain hash size = 128 [ 0.832016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.835103] NetLabel: unlabeled traffic allowed by default [ 0.839241] vgaarb: loaded [ 0.841008] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.843019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.852365] clocksource: Switched to clocksource kvm-clock [ 0.979225] VFS: Disk quotas dquot_6.6.0 [ 0.981073] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.983962] *** VALIDATE ramfs *** [ 0.985530] *** VALIDATE hugetlbfs *** [ 0.987527] pnp: PnP ACPI init [ 0.990030] pnp: PnP ACPI: found 6 devices [ 1.019161] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.022758] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.025124] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.027376] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.030850] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.033746] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.037001] NET: Registered protocol family 2 [ 1.040130] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.045738] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.049723] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.056067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.059933] TCP: Hash tables configured (established 65536 bind 65536) [ 1.063545] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.067316] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070561] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.074724] NET: Registered protocol family 1 [ 1.078152] RPC: Registered named UNIX socket transport module. [ 1.080372] RPC: Registered udp transport module. [ 1.082676] RPC: Registered tcp transport module. [ 1.084872] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.087090] NET: Registered protocol family 44 [ 1.088255] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.091016] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.093348] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.095870] PCI: CLS 0 bytes, default 64 [ 1.097924] Unpacking initramfs... [ 2.476372] debug: unmapping init [mem 0xffff8f48fcc54000-0xffff8f48fffbffff] [ 2.480889] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.487428] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.492847] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.021556] Initialise system trusted keyrings [ 3.023650] Key type blacklist registered [ 3.027497] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.040620] zbud: loaded [ 3.044665] *** VALIDATE nfs *** [ 3.046082] *** VALIDATE nfs4 *** [ 3.048534] pstore: using deflate compression [ 3.052759] Platform Keyring initialized [ 3.186554] NET: Registered protocol family 38 [ 3.189247] Key type asymmetric registered [ 3.190836] Asymmetric key parser 'x509' registered [ 3.192936] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.197415] io scheduler mq-deadline registered [ 3.199085] io scheduler kyber registered [ 3.202352] io scheduler bfq registered [ 3.204251] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.207552] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.210651] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.213293] ACPI: Power Button [PWRF] [ 3.221324] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.229183] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.240772] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.247648] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.262075] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.294415] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.322501] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.328613] Non-volatile memory driver v1.3 [ 3.330462] Linux agpgart interface v0.103 [ 3.367294] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.370586] vda: detected capacity change from 0 to 75243520 [ 3.390550] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.394594] vdb: detected capacity change from 0 to 1073741824 [ 3.415822] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.418707] vdc: detected capacity change from 0 to 2621440000 [ 3.435677] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.438672] vdd: detected capacity change from 0 to 2621440000 [ 3.462348] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.465746] vde: detected capacity change from 0 to 4294967296 [ 3.485388] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.488611] vdf: detected capacity change from 0 to 4294967296 [ 3.497338] libphy: Fixed MDIO Bus: probed [ 3.502168] usbcore: registered new interface driver usbserial_generic [ 3.504896] usbserial: USB Serial support registered for generic [ 3.507304] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.511679] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.513608] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.516489] mousedev: PS/2 mouse device common for all mice [ 3.518671] rtc_cmos 00:05: RTC can wake from S4 [ 3.521366] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.522929] rtc_cmos 00:05: registered as rtc0 [ 3.526954] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.530226] intel_pstate: CPU model not supported [ 3.533043] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.535522] hid: raw HID events driver (C) Jiri Kosina [ 3.540368] usbcore: registered new interface driver usbhid [ 3.543335] usbhid: USB HID core driver [ 3.545238] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.545307] drop_monitor: Initializing network drop monitor service [ 3.551630] Initializing XFRM netlink socket [ 3.553600] NET: Registered protocol family 10 [ 3.556713] Segment Routing with IPv6 [ 3.558604] NET: Registered protocol family 17 [ 3.561409] mpls_gso: MPLS GSO support [ 3.567433] RAS: Correctable Errors collector initialized. [ 3.570297] AVX version of gcm_enc/dec engaged. [ 3.573050] AES CTR mode by8 optimization enabled [ 3.674405] sched_clock: Marking stable (3674379703, 0)->(4567570435, -893190732) [ 3.678966] registered taskstats version 1 [ 3.681592] Loading compiled-in X.509 certificates [ 3.684702] zswap: loaded using pool lzo/zbud [ 3.712967] Key type big_key registered [ 3.727264] Key type encrypted registered [ 3.729485] ima: No TPM chip found, activating TPM-bypass! [ 3.732335] ima: Allocated hash algorithm: sha1 [ 3.734129] ima: No architecture policies found [ 3.736291] evm: Initialising EVM extended attributes: [ 3.738885] evm: security.selinux [ 3.740127] evm: security.ima [ 3.741610] evm: security.capability [ 3.743455] evm: HMAC attrs: 0x1 [ 3.746364] rtc_cmos 00:05: setting system clock to 2026-08-27 22:05:10 UTC (1787868310) [ 3.753992] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 3.757829] debug: unmapping init [mem 0xffffffff88382000-0xffffffff88658fff] [ 3.770093] Write protecting the kernel read-only data: 28672k [ 3.773187] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 3.775893] debug: unmapping init [mem 0xffffffff87314000-0xffffffff873fffff] [ 3.811538] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.820675] systemd[1]: Detected virtualization kvm. [ 3.822618] systemd[1]: Detected architecture x86-64. [ 3.825017] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.855513] systemd[1]: No hostname configured. [ 3.858102] systemd[1]: Set hostname to . [ 3.861434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864326] systemd[1]: Initializing machine ID from random generator. [ 3.911830] random: ln: uninitialized urandom read (6 bytes read) [ 3.998780] random: systemd: uninitialized urandom read (16 bytes read) [ 4.001344] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.005729] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.009593] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.753186] device-mapper: uevent: version 1.0.3 [ 4.755462] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.606876] random: fast init done [ 5.650699] virtio_net virtio0 ens2: renamed from eth0 [ 5.732842] scsi host0: ata_piix [ 5.753934] scsi host1: ata_piix [ 5.755988] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.758573] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.222209] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.254554] random: crng init done [ 10.257200] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.918719] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.235499] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.529823] SELinux: Disabled at runtime. [ 12.592891] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.603780] systemd[1]: Detected virtualization kvm. [ 12.606315] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.260224] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.263305] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.275466] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.280360] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.285649] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.295660] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.303650] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ 13.459953] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.800740] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.122157] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.140940] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.250751] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.290787] EDAC sbridge: Ver: 1.1.2 [ 16.206768] Key type dns_resolver registered [ 16.533424] NFS: Registering the id_resolver key type [ 16.535532] Key type id_resolver registered [ 16.537407] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg636-server login: [ 68.359950] libcfs: loading out-of-tree module taints kernel. [ 68.415377] Key type ._llcrypt registered [ 68.417982] Key type .llcrypt registered [ 68.538976] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 89.592226] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 91.154335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.182220] alg: No test for adler32 (adler32-zlib) [ 93.005359] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 94.252592] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 96.192151] Key type lgssc registered [ 98.507498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.661042] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.166490] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 167.691026] hrtimer: interrupt took 17784533 ns [ 169.245642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.271834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.547108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.579310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.656885] Lustre: lustre-MDT0000: new disk, initializing [ 170.768089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.812835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.258479] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.050115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.147988] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.173272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.177426] Lustre: Skipped 1 previous similar message [ 188.311352] Lustre: lustre-MDT0001: new disk, initializing [ 188.372960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.396686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 188.408570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 193.373908] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.539389] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.575725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.914877] Lustre: lustre-OST0000: new disk, initializing [ 209.924667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.942979] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.027385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.453238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.464494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.550350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.081310] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.683143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.931422] Lustre: lustre-OST0001: new disk, initializing [ 233.942368] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.953499] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.060488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.763693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.795026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.871737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.879639] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.027798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.848687] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.453683] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing check_logdir /tmp/testlogs/ [ 278.855978] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing yml_node [ 282.972566] Lustre: DEBUG MARKER: Client: 2.17.57.99 [ 286.409527] Lustre: DEBUG MARKER: MDS: 2.17.57.99 [ 289.724356] Lustre: DEBUG MARKER: OSS: 2.17.57.99 [ 292.615390] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 18:09:56 EDT 2026 [ 313.178825] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 314.952959] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.752211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.761639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.775318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.308990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.318213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.319059] Lustre: Skipped 1 previous similar message [ 334.359474] Lustre: Skipped 1 previous similar message [ 336.287305] Lustre: server umount lustre-MDT0000 complete [ 340.343887] LustreError: 10335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868647 with bad export cookie 11211931479480303155 [ 340.345273] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.349910] LustreError: 10335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 344.545078] LustreError: 6512:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.545468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.569696] LustreError: 6512:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 344.584848] Lustre: Skipped 1 previous similar message [ 344.585275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.611041] Lustre: Skipped 3 previous similar messages [ 345.569601] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.603774] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 346.908744] Lustre: server umount lustre-MDT0001 complete [ 351.525584] Lustre: server umount lustre-OST0000 complete [ 355.115913] Lustre: server umount lustre-OST0001 complete [ 362.415705] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 375.317428] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 413.897243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.124871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.153890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.241791] Lustre: lustre-MDT0000: new disk, initializing [ 414.343889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.362282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.116556] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 429.805778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.976787] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.058808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.066237] Lustre: Skipped 1 previous similar message [ 430.156916] Lustre: lustre-MDT0001: new disk, initializing [ 430.296262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.342410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.363637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.265519] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 445.333705] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 451.768212] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 459.767780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.271928] Lustre: lustre-OST0000: new disk, initializing [ 460.291951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.302188] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.538306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.883891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 461.895636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.067132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.346869] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 478.530059] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.653107] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.920736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.119315] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.344781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.026896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.032046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.039803] LustreError: Skipped 1 previous similar message [ 493.053930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.067165] Lustre: Skipped 2 previous similar messages [ 493.085470] Lustre: Skipped 3 previous similar messages [ 497.264950] Lustre: server umount lustre-OST0000 complete [ 502.243402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 502.257993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.289253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.128437] Lustre: server umount lustre-MDT0000 complete [ 508.390359] LustreError: 15082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.406035] LustreError: 15082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 510.725988] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868817 with bad export cookie 11211931479480304688 [ 510.730309] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.735449] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 511.369781] Lustre: server umount lustre-MDT0001 complete [ 515.712959] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:13:40 (1787868820) [ 518.315891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 520.168652] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:13:44 (1787868824) [ 526.831905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.353414] LustreError: 19453:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.439156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.276587] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 532.454243] LustreError: 19454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.878650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.992523] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 553.030607] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 558.361812] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 565.629736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.107494] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 580.186548] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 593.889733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.894350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.901076] Lustre: Skipped 1 previous similar message [ 593.904783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.907826] Lustre: Skipped 2 previous similar messages [ 597.635885] Lustre: server umount lustre-OST0000 complete [ 608.543527] Lustre: server umount lustre-MDT0000 complete [ 610.277076] LustreError: 20692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.298704] LustreError: 20692:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 612.953082] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868919 with bad export cookie 11211931479480305570 [ 612.966308] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.981765] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 613.431697] Lustre: server umount lustre-MDT0001 complete [ 624.929918] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 627.777200] Key type lgssc unregistered [ 628.123305] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.136524] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.162217] LNet: Removed LNI 192.168.206.136@tcp [ 629.117160] Key type .llcrypt unregistered [ 629.119383] Key type ._llcrypt unregistered [ 653.912753] Key type ._llcrypt registered [ 653.915663] Key type .llcrypt registered [ 654.049864] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 655.384494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.422818] alg: No test for adler32 (adler32-zlib) [ 656.637590] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 656.967539] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 658.656230] Key type lgssc registered [ 660.030343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.719406] loop: module loaded [ 669.837986] loop0: detected capacity change from 0 to 204800000 [ 670.190259] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.202824] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.542053] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.308526] loop0: detected capacity change from 0 to 204800000 [ 677.459700] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.480741] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.523882] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.218165] loop0: detected capacity change from 0 to 204800000 [ 688.636546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 688.668804] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.973929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 690.050194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 690.172968] Lustre: lustre-MDT0000: new disk, initializing [ 690.282662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.331941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 695.239714] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 703.834081] loop1: detected capacity change from 0 to 204800000 [ 710.132825] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 710.432953] Lustre: lustre-OST07c6: new disk, initializing [ 710.439846] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 710.446543] Lustre: Skipped 1 previous similar message [ 710.455705] Lustre: 26176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 710.527110] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 716.510172] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 718.473903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 718.485338] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 718.524024] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 721.582587] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 721.588542] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 721.592939] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 721.597972] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 721.602942] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 721.607553] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 738.512417] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 738.526740] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 738.534979] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 738.543901] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 738.553826] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 738.563916] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.392441] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 743.402965] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 743.413462] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 743.421104] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.426844] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.433923] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.441815] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 743.447675] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.452589] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 743.456864] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.464385] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.475290] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.656769] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 746.665525] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 746.676477] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 746.683047] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.695943] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.704689] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.711484] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 746.717401] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.722305] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 746.726972] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.733802] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.745573] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.334518] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 751.341389] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 751.348771] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.356043] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.362783] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.370863] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.379401] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 751.391910] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.401175] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 751.410159] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.419118] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 751.427681] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.286919] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.295601] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 762.302974] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.326438] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.339829] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.350600] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.357244] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.361658] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.366672] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.372587] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 762.377633] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.382559] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 810.981950] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 810.995489] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.011356] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 815.433302] Lustre: server umount lustre-OST07c6 complete [ 824.558125] Lustre: server umount lustre-MDT0000 complete [ 841.741295] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 844.259834] Key type lgssc unregistered [ 844.550834] LNet: 28737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.558539] LNetError: 28737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.583259] LNet: Removed LNI 192.168.206.136@tcp [ 845.430479] Key type .llcrypt unregistered [ 845.432630] Key type ._llcrypt unregistered [ 863.120502] Key type ._llcrypt registered [ 863.125845] Key type .llcrypt registered [ 863.240973] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 864.581978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.658901] alg: No test for adler32 (adler32-zlib) [ 865.794270] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 866.145761] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 867.851394] Key type lgssc registered [ 868.876193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.283837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.313620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.970294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.549812] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 887.266806] LustreError: 30008:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.297420] LustreError: 30008:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 890.577471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.806825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.666788] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 900.778101] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.060597] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.639392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.979952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.621121] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 923.257539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 924.556066] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 938.979473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 938.981183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.000403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.005425] Lustre: Skipped 1 previous similar message [ 944.100657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.639540] Lustre: server umount lustre-OST0000 complete [ 949.221912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.254710] Lustre: Skipped 1 previous similar message [ 949.259664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.272376] Lustre: Skipped 2 previous similar messages [ 954.340309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.342951] Lustre: Skipped 1 previous similar message [ 954.901546] Lustre: server umount lustre-MDT0000 complete [ 958.939524] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869265 with bad export cookie 4420018407253582820 [ 958.950056] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.952840] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 959.534904] Lustre: server umount lustre-MDT0001 complete [ 970.307340] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 973.665979] Key type lgssc unregistered [ 974.020084] LNet: 33749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.029762] LNetError: 33749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.046445] LNet: Removed LNI 192.168.206.136@tcp [ 975.007839] Key type .llcrypt unregistered [ 975.010091] Key type ._llcrypt unregistered [ 992.881673] Key type ._llcrypt registered [ 992.885857] Key type .llcrypt registered [ 992.988792] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 994.370617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.399244] alg: No test for adler32 (adler32-zlib) [ 995.412802] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 995.864898] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 997.520241] Key type lgssc registered [ 998.670914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1005.888891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1005.921256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.476972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.590582] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1017.825186] LustreError: 35019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.844055] LustreError: 35019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1019.861076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.257304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.744432] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1032.091762] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1037.162993] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1044.390131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.891828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.824631] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1053.192202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1058.406415] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.596564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.616652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.638099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1073.633653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1073.640875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1073.646846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.664343] Lustre: Skipped 1 previous similar message [ 1075.730977] Lustre: server umount lustre-OST0000 complete [ 1081.826397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1081.839857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.849882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1083.885685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1083.887551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.893769] Lustre: server umount lustre-MDT0000 complete [ 1089.002940] LustreError: 36453:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.022762] LustreError: 36453:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1089.631360] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869396 with bad export cookie 4323342851202565493 [ 1089.638126] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.644522] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1090.012975] Lustre: server umount lustre-MDT0001 complete [ 1100.231756] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1102.770468] Key type lgssc unregistered [ 1103.050293] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.057766] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.106551] LNet: Removed LNI 192.168.206.136@tcp [ 1104.834608] Key type .llcrypt unregistered [ 1104.836987] Key type ._llcrypt unregistered [ 1129.667173] Key type ._llcrypt registered [ 1129.672035] Key type .llcrypt registered [ 1129.791878] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1131.152831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.201824] alg: No test for adler32 (adler32-zlib) [ 1132.309296] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1132.571931] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1134.232098] Key type lgssc registered [ 1135.350978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1143.174116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1143.209400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.792179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.246198] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1155.046473] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.069991] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1157.659730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.023634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1163.082070] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1170.733382] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1176.560672] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1182.873832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1183.248377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.074824] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1189.378930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1196.172265] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1207.785340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.788596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.802840] Lustre: Skipped 2 previous similar messages [ 1207.807338] Lustre: Skipped 1 previous similar message [ 1209.178738] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 1209.185501] Lustre: Skipped 1 previous similar message [ 1212.155495] Lustre: server umount lustre-MDT0000 complete [ 1212.905588] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.924689] LustreError: 39979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1214.285183] LustreError: 39974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.660474] LustreError: 39958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869522 with bad export cookie 1132138359610284645 [ 1215.666280] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.670315] LustreError: 39958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1218.023856] LustreError: 39975:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.024431] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.049122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1220.068599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1220.071048] Lustre: Skipped 1 previous similar message [ 1222.041627] Lustre: server umount lustre-MDT0001 complete [ 1242.451773] Lustre: server umount lustre-OST0000 complete [ 1259.369395] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1262.337530] Key type lgssc unregistered [ 1262.733229] LNet: 43844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.751477] LNetError: 43844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.768798] LNet: Removed LNI 192.168.206.136@tcp [ 1263.630158] Key type .llcrypt unregistered [ 1263.631922] Key type ._llcrypt unregistered [ 1284.679207] Key type ._llcrypt registered [ 1284.681846] Key type .llcrypt registered [ 1284.832197] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1285.744618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1285.855211] alg: No test for adler32 (adler32-zlib) [ 1286.928509] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1287.222451] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1288.968283] Key type lgssc registered [ 1289.963347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1296.957648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1296.997258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.494320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.524628] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1308.650862] LustreError: 45241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.665660] LustreError: 45241:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1310.903086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.378275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1316.223134] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1323.508258] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.966710] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.488759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.707755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.305602] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1343.867287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1351.002741] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1358.305494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.311498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.324985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1359.226207] Lustre: lustre-OST0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 1359.332711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.335244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.167467] Lustre: server umount lustre-OST0000 complete [ 1382.883055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.887911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.906754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.911568] Lustre: Skipped 2 previous similar messages [ 1384.930042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.930568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.962469] Lustre: Skipped 1 previous similar message [ 1388.005530] Lustre: server umount lustre-MDT0000 complete [ 1390.058515] LustreError: 45259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.082451] LustreError: 45259:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1392.007822] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869698 with bad export cookie 16921245481824034978 [ 1392.019462] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.400943] Lustre: server umount lustre-MDT0001 complete [ 1408.529922] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1410.722638] Key type lgssc unregistered [ 1410.969102] LNet: 48952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.975868] LNetError: 48952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.004047] LNet: Removed LNI 192.168.206.136@tcp [ 1411.858771] Key type .llcrypt unregistered [ 1411.861151] Key type ._llcrypt unregistered [ 1430.591747] Key type ._llcrypt registered [ 1430.593383] Key type .llcrypt registered [ 1430.686619] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1431.498177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.537895] alg: No test for adler32 (adler32-zlib) [ 1432.530881] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1432.751382] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1434.400218] Key type lgssc registered [ 1435.211192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1440.918247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1440.934459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.252825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.863451] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1451.500994] LustreError: 50348:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.516371] LustreError: 50348:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1453.317172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.614302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.808685] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1464.609580] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1469.924365] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.384329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.749742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.673862] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1486.974362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1490.520267] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.635821] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1498.853511] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1511.004092] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2718h30m15s [ 1512.344979] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1514.980835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1514.998720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.019826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.069640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.087712] Lustre: Skipped 4 previous similar messages [ 1517.914392] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 1520.850558] Lustre: server umount lustre-MDT0000 complete [ 1522.167554] LustreError: 50348:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.210214] LustreError: 50348:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1524.048741] LustreError: 50343:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.458460] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1528.308850] LustreError: 51386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.343766] LustreError: 51386:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1531.308205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.486709] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.807118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.869475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1532.858504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1535.178971] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1538.702722] Lustre: DEBUG MARKER: Wait for df (36756) ... [ 1540.450747] Lustre: DEBUG MARKER: done [ 1545.697285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1545.704953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.725623] Lustre: Skipped 2 previous similar messages [ 1545.732451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.738378] Lustre: Skipped 1 previous similar message [ 1547.232851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1547.243656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.253263] Lustre: server umount lustre-OST0000 complete [ 1558.497078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.499922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.528465] Lustre: Skipped 1 previous similar message [ 1558.552678] Lustre: Skipped 4 previous similar messages [ 1560.018468] Lustre: server umount lustre-MDT0000 complete [ 1563.724583] LustreError: 52975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869870 with bad export cookie 15466306319360476138 [ 1563.727402] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1563.739483] LustreError: 52975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1564.071181] Lustre: server umount lustre-MDT0001 complete [ 1573.362639] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1575.744156] Key type lgssc unregistered [ 1576.015545] LNet: 55274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.026719] LNetError: 55274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.045188] LNet: Removed LNI 192.168.206.136@tcp [ 1576.853994] Key type .llcrypt unregistered [ 1576.856814] Key type ._llcrypt unregistered [ 1613.437613] Key type ._llcrypt registered [ 1613.440593] Key type .llcrypt registered [ 1613.549038] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1614.632118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1614.683179] alg: No test for adler32 (adler32-zlib) [ 1615.687248] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1615.879841] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1617.528154] Key type lgssc registered [ 1618.568805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1624.885062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1624.901076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.148606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1626.171084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1626.325373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.646928] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1639.162663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.289754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1639.346424] Lustre: 57463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1639.631959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1644.134027] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1650.932531] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1656.590263] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1662.984530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1663.251055] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1663.487647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.960201] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1670.274450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1677.076240] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1684.357736] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2718h33m09s [ 1685.981355] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1692.391394] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1694.629384] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1694.700456] LustreError: 57471:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1715.059710] Lustre: lustre-MDT0000: Client 20f5e6ec-1bae-4159-a9e3-77e506065a99 (at 192.168.206.36@tcp) reconnecting [ 1734.769289] LustreError: 57471:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1736.877215] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1739.001611] Lustre: DEBUG MARKER: done [ 1746.915182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.921980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.936465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.427322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.447952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.469938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.473514] Lustre: Skipped 1 previous similar message [ 1750.030229] Lustre: server umount lustre-OST0000 complete [ 1757.665945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1757.669817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.678396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.467491] Lustre: server umount lustre-MDT0000 complete [ 1762.870969] LustreError: 57457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870069 with bad export cookie 8909947043666992275 [ 1762.881093] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.418779] Lustre: server umount lustre-MDT0001 complete [ 1772.919758] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1775.170755] Key type lgssc unregistered [ 1775.474836] LNet: 61795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.486318] LNetError: 61795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.504569] LNet: Removed LNI 192.168.206.136@tcp [ 1776.136855] Key type .llcrypt unregistered [ 1776.139375] Key type ._llcrypt unregistered [ 1817.815704] Key type ._llcrypt registered [ 1817.820137] Key type .llcrypt registered [ 1817.928402] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1819.123756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.149214] alg: No test for adler32 (adler32-zlib) [ 1820.219159] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1820.459208] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1822.152176] Key type lgssc registered [ 1823.434338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.457749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.485263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.767532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1827.805362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1827.985708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.860052] Lustre: Failing over lustre-MDT0000 [ 1829.053334] Lustre: server umount lustre-MDT0000 complete [ 1834.748778] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:35:39 (1787870139) [ 1840.071373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.418784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.776721] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1851.244615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.335517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1851.366563] Lustre: 64598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1851.605639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.673785] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1863.516869] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1869.787662] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1876.743092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1876.873306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1877.012667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.031718] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1884.797511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1887.900364] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.654409] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1900.521605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.527468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.542040] Lustre: Skipped 1 previous similar message [ 1900.552220] Lustre: Skipped 1 previous similar message [ 1905.029132] Lustre: server umount lustre-MDT0000 complete [ 1908.621292] LustreError: 64590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870215 with bad export cookie 12179113770161528403 [ 1908.626517] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.634266] LustreError: 64590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1908.938985] Lustre: server umount lustre-MDT0001 complete [ 1912.203903] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1916.089663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1923.324867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.136255] Lustre: 63523:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870217/real 1787870217] req@0000000095176101 x1874717706327040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1927.168960] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.208200] Lustre: 63525:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870220/real 1787870220] req@000000006d01c7b4 x1874717706327168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1932.257485] Lustre: 63525:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870222/real 1787870222] req@0000000033ffa5ff x1874717706327296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1933.344901] LustreError: 63521:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001c1e378b x1874717706328064/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1933.680525] LustreError: 66444:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.691682] LustreError: 66444:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1933.774173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.856302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1934.752089] Lustre: 63523:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870225/real 1787870225] req@000000000fe0933c x1874717706327552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1935.781204] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.808698] LustreError: 68366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.838575] LustreError: 68366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1937.473410] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1939.938593] Lustre: 63523:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870230/real 1787870230] req@00000000ae433310 x1874717706327680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1941.987498] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.016501] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1945.331144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.687675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1949.814487] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1950.699398] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1955.876405] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1957.328474] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1962.789738] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1971.169800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.176423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.179534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.210171] Lustre: Skipped 2 previous similar messages [ 1975.720871] Lustre: server umount lustre-MDT0000 complete [ 1976.290773] LustreError: 66444:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.333177] LustreError: 66444:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1979.463798] LustreError: 64588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870286 with bad export cookie 12179113770161530685 [ 1979.469945] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.473232] LustreError: 64588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.806422] Lustre: server umount lustre-MDT0001 complete [ 1985.866131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1994.507934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.794110] Lustre: 63524:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870288/real 1787870288] req@0000000085793ec4 x1874717706353152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1997.825899] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.042523] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa904e7723d8beca0 [ 2005.048022] Lustre: MGC192.168.206.136@tcp: Connection restored to 0@lo (at 0@lo) [ 2005.433657] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.484476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.546598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2010.298651] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2020.064027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.921868] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2025.968428] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.138836] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.170456] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.926685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2045.932991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.948351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.957204] Lustre: Skipped 2 previous similar messages [ 2050.760178] Lustre: server umount lustre-MDT0000 complete [ 2052.579772] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.591344] LustreError: 66445:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 2054.356728] LustreError: 66451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870361 with bad export cookie 12179113770161532064 [ 2054.357977] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.366872] LustreError: 66451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2054.610339] Lustre: server umount lustre-MDT0001 complete [ 2060.189512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2063.404878] Lustre: DEBUG MARKER: files compared the same [ 2075.262675] Lustre: server umount lustre-OST0000 complete [ 2092.269391] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2095.322297] Key type lgssc unregistered [ 2095.663318] LNet: 74325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.680481] LNetError: 74325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.698826] LNet: Removed LNI 192.168.206.136@tcp [ 2096.615952] Key type .llcrypt unregistered [ 2096.618377] Key type ._llcrypt unregistered [ 2119.754078] Key type ._llcrypt registered [ 2119.760459] Key type .llcrypt registered [ 2119.868453] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2121.186940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.243074] alg: No test for adler32 (adler32-zlib) [ 2122.355482] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2122.648219] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2124.376172] Key type lgssc registered [ 2125.477576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.095985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.111962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.616202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.809361] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug +malloc all [ 2144.226705] LustreError: 75742:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.242628] LustreError: 75742:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2144.376989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.571913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.148174] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug +malloc all [ 2155.636373] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2159.827256] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2166.122426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2166.365577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.371677] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug +malloc all [ 2177.842413] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2178.667900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2189.287030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2189.292193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.293410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.301759] Lustre: Skipped 1 previous similar message [ 2189.315329] Lustre: Skipped 1 previous similar message [ 2194.403494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2194.417970] Lustre: Skipped 1 previous similar message [ 2194.737443] Lustre: server umount lustre-OST0000 complete [ 2199.521382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.523128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.541166] Lustre: Skipped 1 previous similar message [ 2199.549054] Lustre: Skipped 1 previous similar message [ 2203.865674] Lustre: server umount lustre-MDT0000 complete [ 2204.642072] LustreError: 75743:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.655091] LustreError: 75743:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2207.116236] LustreError: 75722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870513 with bad export cookie 1928295148708957534 [ 2207.121280] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.122971] LustreError: 75722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2207.385762] Lustre: server umount lustre-MDT0001 complete [ 2216.713952] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2219.017301] Key type lgssc unregistered [ 2219.271141] LNet: 79276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.276496] LNetError: 79276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.289016] LNet: Removed LNI 192.168.206.136@tcp [ 2220.059814] Key type .llcrypt unregistered [ 2220.065290] Key type ._llcrypt unregistered [ 2244.374623] Key type ._llcrypt registered [ 2244.377660] Key type .llcrypt registered [ 2244.480516] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2245.529659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.558483] alg: No test for adler32 (adler32-zlib) [ 2246.743296] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2246.930525] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2248.648202] Key type lgssc registered [ 2249.975561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.536639] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.557857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.056368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2278.176423] LustreError: 80667:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 2304.800348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.337301] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2316.638672] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.315938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.427935] Lustre: *** cfs_fail_loc=706, val=0*** [ 2323.751398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.798433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2327.728749] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2329.061315] LustreError: 80693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.072183] LustreError: 80693:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2334.182368] LustreError: 80692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.249551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.569083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.283096] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2348.264432] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2353.643390] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.194224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.205768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.222075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.785358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.806483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.823610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.825255] Lustre: server umount lustre-OST0000 complete [ 2365.411794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.418138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.422885] Lustre: Skipped 1 previous similar message [ 2365.431142] Lustre: Skipped 3 previous similar messages [ 2370.530955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.545195] Lustre: Skipped 1 previous similar message [ 2371.670554] Lustre: server umount lustre-MDT0000 complete [ 2374.636848] LustreError: 80673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870681 with bad export cookie 14274824771298983561 [ 2374.643640] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.647126] LustreError: 80673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2374.946594] Lustre: server umount lustre-MDT0001 complete [ 2383.846746] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2386.418382] Key type lgssc unregistered [ 2386.661599] LNet: 84283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.682569] LNetError: 84283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.751911] LNet: Removed LNI 192.168.206.136@tcp [ 2388.589204] Key type .llcrypt unregistered [ 2388.592257] Key type ._llcrypt unregistered [ 2411.377825] Key type ._llcrypt registered [ 2411.380248] Key type .llcrypt registered [ 2411.443793] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2412.231875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.265202] alg: No test for adler32 (adler32-zlib) [ 2413.265935] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2413.409859] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2415.040685] Key type lgssc registered [ 2415.922757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2422.610888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.630271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.660335] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2435.848283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.447583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.075334] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2441.700590] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.727192] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2446.820092] LustreError: 86392:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.837920] LustreError: 86392:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2448.800944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.155541] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.196374] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2449.205390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.498079] LustreError: 87135:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.510686] LustreError: 87135:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2455.091362] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2459.619732] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.641749] LustreError: 86388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2462.941488] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2462.975665] Lustre: MGS: Client 218a93c7-a6ac-4f08-af19-410cff9aed7c (at 0@lo) reconnecting [ 2463.005287] LustreError: 86402:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2463.257885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.347802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2467.614718] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2480.098885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2480.107496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.131900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.688110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.689034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2483.699447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.231343] Lustre: server umount lustre-OST0000 complete [ 2493.924409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.930240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.931887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.931896] Lustre: Skipped 1 previous similar message [ 2493.939497] Lustre: Skipped 1 previous similar message [ 2495.760339] Lustre: server umount lustre-MDT0000 complete [ 2499.173757] Lustre: server umount lustre-MDT0001 complete [ 2502.642178] Lustre: server umount MGS complete [ 2513.664981] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:46:58 (1787870818) [ 2542.874737] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 2552.495936] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2587.178877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.794869] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2600.884614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.125812] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2601.178648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2601.257642] Lustre: lustre-MDT0001: new disk, initializing [ 2601.343815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.381035] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2601.394193] Lustre: 93553:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2601.401747] LustreError: 93553:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2605.556487] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2614.712281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.898396] Lustre: lustre-OST0000: new disk, initializing [ 2614.901691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2614.907093] Lustre: 94319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2614.966513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.468967] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2624.544978] Lustre: 94819:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000022ba418a x1874718328134656/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787870947 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2628.602593] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2635.017928] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.031069] Lustre: MGS: Client 18d6aec7-906e-4923-96d0-c397ca6cb793 (at 0@lo) reconnecting [ 2635.083857] Lustre: 95365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2635.098520] LustreError: 93539:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2635.130616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2635.224762] Lustre: lustre-MDT0000: new disk, initializing [ 2635.417317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.463051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2635.587313] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2636.513201] Lustre: 95390:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000093302433 x1874718328141056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787870959 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2639.822614] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2644.967780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2644.976423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2645.676199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2645.813180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2669.542873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.546123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.558566] Lustre: Skipped 1 previous similar message [ 2669.570850] Lustre: Skipped 2 previous similar messages [ 2674.375403] Lustre: server umount lustre-OST0000 complete [ 2679.780471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.793861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.804177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.264805] Lustre: server umount lustre-MDT0000 complete [ 2684.898022] LustreError: 93525:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.917023] LustreError: 93525:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2693.676698] Lustre: server umount lustre-MDT0001 complete [ 2697.269769] Lustre: server umount MGS complete [ 2706.814805] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:50:11 (1787871011) [ 2720.355513] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2722.668773] Key type lgssc unregistered [ 2722.939568] LNet: 97888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.945799] LNetError: 97888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.963037] LNet: Removed LNI 192.168.206.136@tcp [ 2723.735895] Key type .llcrypt unregistered [ 2723.742868] Key type ._llcrypt unregistered [ 2737.552959] Key type ._llcrypt registered [ 2737.555341] Key type .llcrypt registered [ 2737.735442] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2738.569703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2738.687362] alg: No test for adler32 (adler32-zlib) [ 2739.664906] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2739.838555] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2741.464335] Key type lgssc registered [ 2742.420472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2749.274970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2749.297064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.755256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.540583] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2761.188352] LustreError: 99090:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.220994] LustreError: 99090:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2762.025366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.287507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.963929] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2774.044087] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2780.769811] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2788.381862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.789117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.290248] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2795.951766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2801.783599] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2812.391906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.408784] Lustre: Skipped 1 previous similar message [ 2812.409278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.113906] Lustre: server umount lustre-OST0000 complete [ 2817.505865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.518607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.527376] Lustre: Skipped 1 previous similar message [ 2817.545351] Lustre: Skipped 2 previous similar messages [ 2822.630676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.637115] Lustre: Skipped 1 previous similar message [ 2823.616782] Lustre: server umount lustre-MDT0000 complete [ 2826.860487] LustreError: 99070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871133 with bad export cookie 6684370072285758478 [ 2826.869378] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.872329] LustreError: 99070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2827.152537] Lustre: server umount lustre-MDT0001 complete [ 2837.869582] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2840.794973] Key type lgssc unregistered [ 2841.131615] LNet: 102628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2841.144077] LNetError: 102628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2841.159568] LNet: Removed LNI 192.168.206.136@tcp [ 2842.117972] Key type .llcrypt unregistered [ 2842.123751] Key type ._llcrypt unregistered [ 2858.323280] Key type ._llcrypt registered [ 2858.326427] Key type .llcrypt registered [ 2858.446422] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2859.339626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2859.495740] alg: No test for adler32 (adler32-zlib) [ 2860.557611] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2860.766994] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2862.449467] Key type lgssc registered [ 2863.771321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2869.934162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2869.953789] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2869.980714] LustreError: 103715:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2869.986605] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 2869.993128] LustreError: 103707:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2870.007768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.017284] LustreError: 103707:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2871.470405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.481708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.877588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2884.475561] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2890.974427] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing lsmod [ 2894.096949] LustreError: 104916:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2894.133329] LustreError: 104923:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2894.144505] LustreError: 104916:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 2894.160508] LustreError: 104916:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2894.171638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.175437] LustreError: 104916:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2894.654370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.696991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2900.452581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.459896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.481320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.993382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.995694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.013336] Lustre: Skipped 1 previous similar message [ 2902.034393] Lustre: Skipped 2 previous similar messages [ 2903.415017] Lustre: server umount lustre-MDT0000 complete [ 2905.134794] LustreError: 103748:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.164634] LustreError: 103748:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2907.548388] LustreError: 103725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871214 with bad export cookie 3682523206464334220 [ 2907.560088] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.346478] LustreError: 104183:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000095176101 x1874718796829952/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2909.569628] Lustre: server umount lustre-MDT0001 complete [ 2923.762577] Lustre: server umount lustre-OST0000 complete [ 2931.877029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.375726] LustreError: 105985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.470697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.471284] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2937.827409] LustreError: 105986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.952759] LustreError: 105985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.556956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.897350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2950.638616] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2958.348722] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2963.368221] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2969.874648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.350282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.883316] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2977.529579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2983.449986] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2992.097210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.102425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.115120] LustreError: 107810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.200903] Lustre: server umount lustre-OST0000 complete [ 2997.221367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2997.226709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.252146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.361973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.369327] Lustre: Skipped 3 previous similar messages [ 3005.472710] Lustre: server umount lustre-MDT0000 complete [ 3007.460747] LustreError: 105985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.479433] LustreError: 105985:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3008.929511] LustreError: 105967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871315 with bad export cookie 3682523206464335165 [ 3008.938127] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.949400] LustreError: 105967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3009.171764] Lustre: server umount lustre-MDT0001 complete [ 3021.615717] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3024.800345] Key type lgssc unregistered [ 3025.063695] LNet: 109521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.069384] LNetError: 109521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.097512] LNet: Removed LNI 192.168.206.136@tcp [ 3026.458499] Key type .llcrypt unregistered [ 3026.461466] Key type ._llcrypt unregistered [ 3050.787132] Key type ._llcrypt registered [ 3050.788802] Key type .llcrypt registered [ 3050.880963] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3051.560850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3051.626337] alg: No test for adler32 (adler32-zlib) [ 3052.674494] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 3052.892811] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3054.528196] Key type lgssc registered [ 3055.591482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3063.078975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3063.095232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.677792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.376416] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3075.043785] LustreError: 110937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.066319] LustreError: 110937:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3078.037466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.675549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.634623] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3088.862984] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.584575] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.186256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.424189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.094653] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3109.687576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3110.194802] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.256430] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3117.009984] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3124.194494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3124.204066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.222791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.222181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.221073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.244302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3125.253522] Lustre: Skipped 1 previous similar message [ 3126.754208] Lustre: server umount lustre-OST0000 complete [ 3130.337164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.340813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.353960] Lustre: Skipped 1 previous similar message [ 3135.457390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.467274] Lustre: Skipped 2 previous similar messages [ 3136.545624] Lustre: server umount lustre-MDT0000 complete [ 3139.831123] LustreError: 110917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871446 with bad export cookie 17021537825306342751 [ 3139.838661] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.847232] LustreError: 110917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3140.135842] Lustre: server umount lustre-MDT0001 complete [ 3149.999574] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3152.966410] Key type lgssc unregistered [ 3153.325745] LNet: 114576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3153.341771] LNetError: 114576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3153.373287] LNet: Removed LNI 192.168.206.136@tcp [ 3154.293801] Key type .llcrypt unregistered [ 3154.300036] Key type ._llcrypt unregistered [ 3170.387964] Key type ._llcrypt registered [ 3170.390712] Key type .llcrypt registered [ 3170.492412] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3171.589914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.626717] alg: No test for adler32 (adler32-zlib) [ 3172.656137] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 3172.901137] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3174.592164] Key type lgssc registered [ 3175.743314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3182.859673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3182.880991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.327992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.794371] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3194.853689] LustreError: 115801:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.882259] LustreError: 115801:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3195.634017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.933609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.286883] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3210.219974] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3217.321484] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3223.704677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.974895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.553887] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3230.133797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3237.386837] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3250.148482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3250.152422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.179198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.660104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.667883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.680546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.550586] Lustre: server umount lustre-OST0000 complete [ 3255.778150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.793640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.800241] Lustre: Skipped 1 previous similar message [ 3260.897751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.900950] Lustre: Skipped 1 previous similar message [ 3261.278870] Lustre: server umount lustre-MDT0000 complete [ 3264.825894] LustreError: 115781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871571 with bad export cookie 9027351336487332064 [ 3264.833995] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.846563] LustreError: 115781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3265.312616] Lustre: server umount lustre-MDT0001 complete [ 3276.178735] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3278.878041] Key type lgssc unregistered [ 3279.310735] LNet: 119352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3279.324180] LNetError: 119352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3279.355824] LNet: Removed LNI 192.168.206.136@tcp [ 3280.400754] Key type .llcrypt unregistered [ 3280.407342] Key type ._llcrypt unregistered [ 3306.909513] Key type ._llcrypt registered [ 3306.910984] Key type .llcrypt registered [ 3306.966933] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3307.576130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3307.633535] alg: No test for adler32 (adler32-zlib) [ 3308.614619] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 3308.750705] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3310.376889] Key type lgssc registered [ 3311.879374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3320.704507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3320.729036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.239934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.222606] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3331.557287] LustreError: 120768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.577623] LustreError: 120768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3334.808604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.260537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.007481] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3348.981641] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.805310] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3361.469734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.846664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.951037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3368.095239] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3375.480939] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3385.620711] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3389.075319] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3389.079968] Lustre: Skipped 2 previous similar messages [ 3393.052926] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3393.059447] Lustre: Skipped 2 previous similar messages [ 3395.730064] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3395.767114] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3399.811852] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3399.823903] Lustre: Skipped 2 previous similar messages [ 3404.878764] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.36@tcp 192.168.0.[2,10]@tcp [ 3409.778698] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.36/24@tcp in log lustre-MDT0000 [ 3409.783355] Lustre: Skipped 2 previous similar messages [ 3414.574385] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.36/24@tcp [ 3414.586662] Lustre: Skipped 1 previous similar message [ 3421.159848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3421.168129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.171466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.196100] Lustre: Skipped 1 previous similar message [ 3423.309258] Lustre: server umount lustre-OST0000 complete [ 3427.297263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3427.320072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.334265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.339238] Lustre: Skipped 1 previous similar message [ 3431.407847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.416848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.004718] Lustre: server umount lustre-MDT0000 complete [ 3436.515568] LustreError: 122512:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.541458] LustreError: 122512:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3436.658855] LustreError: 120750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871743 with bad export cookie 1313385820754599155 [ 3436.666167] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.676445] LustreError: 120750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3437.172522] Lustre: server umount lustre-MDT0001 complete [ 3451.544245] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3455.495745] Key type lgssc unregistered [ 3455.997446] LNet: 125668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.011725] LNetError: 125668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3456.029547] LNet: Removed LNI 192.168.206.136@tcp [ 3457.021644] Key type .llcrypt unregistered [ 3457.023412] Key type ._llcrypt unregistered [ 3485.689768] Key type ._llcrypt registered [ 3485.693600] Key type .llcrypt registered [ 3485.815959] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3486.457816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3486.547205] alg: No test for adler32 (adler32-zlib) [ 3487.645812] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 3487.928529] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3489.632431] Key type lgssc registered [ 3490.740681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3497.466761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3497.484918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.843142] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3498.861174] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.36/24@tcp [ 3498.884672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.130073] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3509.218404] LustreError: 127243:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.239391] LustreError: 127243:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3510.760728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.037408] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3511.073937] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.36/24@tcp [ 3511.134496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.095505] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3522.925778] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3529.537808] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3536.626976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.990553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.936694] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3544.199104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3550.413868] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3566.049748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.063548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.081981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.096692] Lustre: Skipped 1 previous similar message [ 3566.114296] Lustre: Skipped 1 previous similar message [ 3571.170279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.180190] Lustre: Skipped 1 previous similar message [ 3571.742881] Lustre: server umount lustre-OST0000 complete [ 3576.295030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.296322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3576.333977] Lustre: Skipped 1 previous similar message [ 3576.369043] Lustre: Skipped 1 previous similar message [ 3581.414481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3581.827319] Lustre: server umount lustre-MDT0000 complete [ 3586.533672] LustreError: 127240:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.577982] LustreError: 127240:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3587.021823] LustreError: 127223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871893 with bad export cookie 11981552314575450025 [ 3587.030605] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.045781] LustreError: 127223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3587.661335] Lustre: server umount lustre-MDT0001 complete [ 3600.519340] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3603.919456] Key type lgssc unregistered [ 3604.444841] LNet: 130835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3604.462951] LNetError: 130835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3604.500262] LNet: Removed LNI 192.168.206.136@tcp [ 3605.784786] Key type .llcrypt unregistered [ 3605.788985] Key type ._llcrypt unregistered [ 3633.285386] Key type ._llcrypt registered [ 3633.287676] Key type .llcrypt registered [ 3633.394333] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3647.109497] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3648.188075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3648.315125] alg: No test for adler32 (adler32-zlib) [ 3649.342730] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 3649.582723] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 3651.224169] Key type lgssc registered [ 3652.338113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3689.891598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.934547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.226402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3691.274103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3691.418727] Lustre: lustre-MDT0000: new disk, initializing [ 3691.564150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.601937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3695.762237] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3706.856305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.025281] Lustre: 134234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3707.121288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3707.129262] Lustre: Skipped 1 previous similar message [ 3707.259493] Lustre: lustre-MDT0001: new disk, initializing [ 3707.397159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3707.435284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3707.451823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3712.126309] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3722.468512] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3728.680542] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3736.440194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.831165] Lustre: lustre-OST0000: new disk, initializing [ 3736.834433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3736.844292] Lustre: 136201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3736.967142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.321936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3742.338365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3742.445673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3743.302392] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3751.853405] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3756.331075] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3756.554858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3761.738068] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3761.947342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3768.294284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.295808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.314376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.322573] Lustre: Skipped 1 previous similar message [ 3770.164417] Lustre: server umount lustre-OST0000 complete [ 3773.921813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3773.936152] LustreError: Skipped 1 previous similar message [ 3773.946314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.962241] Lustre: Skipped 1 previous similar message [ 3773.969964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.531907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.533480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.009846] Lustre: server umount lustre-MDT0000 complete [ 3783.654473] LustreError: 135029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872090 with bad export cookie 17231564482520089036 [ 3783.661250] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.675273] LustreError: 135029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3784.103048] Lustre: server umount lustre-MDT0001 complete [ 3792.962745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.388798] LustreError: 138418:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.406954] LustreError: 138418:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3793.441700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.223615] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3798.499666] LustreError: 138419:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.620315] LustreError: 138418:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.934316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.301556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3810.909803] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3818.208028] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3824.303278] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3830.924957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3831.312342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.743237] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3845.628895] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3850.984888] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3851.192188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3855.838558] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3856.046498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3865.685904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.885405] Lustre: lustre-OST0001: new disk, initializing [ 3865.888853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3865.894309] Lustre: 141792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.992152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3871.842229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3871.868862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3871.985315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3872.990882] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3881.500145] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3881.709949] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3886.408473] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3886.630236] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3892.807939] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4d8470d000.ost_server_uuid 50 [ 3894.444546] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4d8470d000.ost_server_uuid in IDLE state after 0 sec [ 3906.406235] Lustre: server umount lustre-OST0001 complete [ 3908.065234] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.072822] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.087901] LustreError: 140241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.111653] Lustre: Skipped 1 previous similar message [ 3908.147321] LustreError: 140241:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3912.164197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3912.182481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.199585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.203426] Lustre: Skipped 1 previous similar message [ 3913.196276] LustreError: 140241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.221338] LustreError: 140241:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3917.440540] Lustre: server umount lustre-OST0000 complete [ 3923.438493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.459107] Lustre: Skipped 1 previous similar message [ 3923.476315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.484079] Lustre: Skipped 3 previous similar messages [ 3928.325479] Lustre: server umount lustre-MDT0000 complete [ 3928.547055] LustreError: 138415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.563821] LustreError: 138415:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3931.795892] LustreError: 138398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872238 with bad export cookie 17231564482520089918 [ 3931.810354] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.817612] LustreError: 138398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3932.102734] Lustre: server umount lustre-MDT0001 complete [ 3953.381491] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 3956.422679] Key type lgssc unregistered [ 3956.690679] LNet: 144796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3956.699649] LNetError: 144796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3956.720337] LNet: Removed LNI 192.168.206.136@tcp [ 3957.755268] Key type .llcrypt unregistered [ 3957.761421] Key type ._llcrypt unregistered [ 3998.151587] Key type ._llcrypt registered [ 3998.153547] Key type .llcrypt registered [ 3998.247021] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 4012.598659] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4013.784896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4013.813894] alg: No test for adler32 (adler32-zlib) [ 4014.976885] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 4015.223892] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 4016.920164] Key type lgssc registered [ 4018.115692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4053.054203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4053.072045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.333521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4054.375423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4054.512246] Lustre: lustre-MDT0000: new disk, initializing [ 4054.615062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.655621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4059.582638] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4069.155507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.271381] Lustre: 148953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4069.398876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4069.408246] Lustre: Skipped 1 previous similar message [ 4069.574713] Lustre: lustre-MDT0001: new disk, initializing [ 4069.680161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4069.736719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4069.756619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4074.810588] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4084.716923] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.206700] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4098.777837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.179771] Lustre: lustre-OST0000: new disk, initializing [ 4099.184795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4099.195564] Lustre: 150920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4099.383104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.445917] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4107.922904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4107.931805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.041993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4118.274974] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4122.599143] Lustre: Failing over lustre-OST0000 [ 4122.924819] Lustre: server umount lustre-OST0000 complete [ 4140.000436] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872430/real 1787872430] req@000000006a4b242a x1874720006898688/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787872446 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4140.003164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.058483] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4141.090749] Lustre: 147176:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872431/real 1787872431] req@000000005d5ad19d x1874720006899456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787872447 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4142.168368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.553991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.574984] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4144.608761] Lustre: 147176:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872435/real 1787872435] req@00000000d214a92e x1874720006900480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787872451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4149.256082] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4149.665289] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872440/real 1787872440] req@00000000a67a711e x1874720006901760/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787872456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4149.687024] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4149.694225] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4152.167818] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4152.168058] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4153.301317] Lustre: Failing over lustre-MDT0000 [ 4153.606523] Lustre: server umount lustre-MDT0000 complete [ 4154.857531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.878219] Lustre: Skipped 3 previous similar messages [ 4155.216163] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872445/real 1787872445] req@0000000051d8e5bf x1874720006902784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787872461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4155.247910] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4159.970714] LustreError: 151843:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.979864] LustreError: 151843:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4162.378444] LustreError: 148957:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.091066] LustreError: 148960:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.123835] LustreError: 148960:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4167.511953] LustreError: 148959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.233420] Lustre: 147178:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872461/real 1787872461] req@00000000ed721394 x1874720006912640/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787872477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4171.261819] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.535897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.615932] LustreError: 148957:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.650771] LustreError: 148957:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4181.478980] LustreError: 148963:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.502821] LustreError: 148963:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4181.696564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.754637] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4182.856246] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4186.256499] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4187.125277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4187.141034] Lustre: Skipped 1 previous similar message [ 4187.171792] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4187.243971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4197.358135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.360516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.395249] Lustre: Skipped 1 previous similar message [ 4197.407990] Lustre: Skipped 1 previous similar message [ 4201.071393] Lustre: server umount lustre-OST0000 complete [ 4207.586134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.588403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4207.589175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.615676] Lustre: Skipped 1 previous similar message [ 4212.711747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.731723] Lustre: Skipped 2 previous similar messages [ 4217.828788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.841655] Lustre: Skipped 1 previous similar message [ 4219.361225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.689190] Lustre: server umount lustre-MDT0000 complete [ 4222.948957] LustreError: 148963:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.967670] LustreError: 148963:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4224.528781] LustreError: 148945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872531 with bad export cookie 14013704046191262617 [ 4224.533336] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.542799] LustreError: 148945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4225.185604] Lustre: server umount lustre-MDT0001 complete [ 4238.802322] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 4242.205341] Key type lgssc unregistered [ 4242.592258] LNet: 154585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4242.614253] LNetError: 154585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4242.633750] LNet: Removed LNI 192.168.206.136@tcp [ 4243.520823] Key type .llcrypt unregistered [ 4243.522915] Key type ._llcrypt unregistered [ 4284.045621] Key type ._llcrypt registered [ 4284.049138] Key type .llcrypt registered [ 4284.191567] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4285.426601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4285.533284] alg: No test for adler32 (adler32-zlib) [ 4286.807274] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 4286.971932] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 4288.688533] Key type lgssc registered [ 4290.274838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4298.470749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4298.509680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.052355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.095505] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4309.481980] LustreError: 156105:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.491094] LustreError: 156105:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4314.327879] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4314.596130] LustreError: 156106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.308588] LustreError: 156134:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4318.331832] LustreError: 156134:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4318.339302] LustreError: 156134:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4318.600995] Lustre: server umount lustre-MDT0000 complete [ 4332.766449] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 4341.457483] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4380.323982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.635624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4380.696845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4380.808512] Lustre: lustre-MDT0000: new disk, initializing [ 4380.923562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.939799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4385.976551] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4396.536984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.649895] Lustre: 159441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4396.700116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4396.704119] Lustre: Skipped 1 previous similar message [ 4396.794388] Lustre: lustre-MDT0001: new disk, initializing [ 4396.903481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.948767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4396.959089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4401.640550] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4409.734521] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4414.808339] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4420.691535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.894701] Lustre: lustre-OST0000: new disk, initializing [ 4420.903203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4420.913313] Lustre: 161410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4421.008953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.169735] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4428.861810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4428.873212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4428.957849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4435.765833] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4439.765372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4441.517160] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:19:06 (1787872746) [ 4705.767077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.768333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.789991] Lustre: Skipped 2 previous similar messages [ 4705.802827] Lustre: Skipped 2 previous similar messages [ 4710.886371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.904139] Lustre: Skipped 2 previous similar messages [ 4711.433205] Lustre: server umount lustre-MDT0000 complete [ 4715.108119] LustreError: 160245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873021 with bad export cookie 4004225059975602210 [ 4715.108811] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.123544] LustreError: 160245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4716.004307] LustreError: 159447:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.004552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.037689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4720.044887] LustreError: 159449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.045403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4721.389716] Lustre: server umount lustre-MDT0001 complete [ 4731.243046] Lustre: server umount lustre-OST0000 complete [ 4740.078772] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 4751.371653] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4789.393743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.658291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.682174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4789.769831] Lustre: lustre-MDT0000: new disk, initializing [ 4789.848578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.866264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4793.822503] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4804.138836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.261506] Lustre: 166075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4804.320662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4804.331852] Lustre: Skipped 1 previous similar message [ 4804.449215] Lustre: lustre-MDT0001: new disk, initializing [ 4804.558695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4804.601318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4804.617681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4808.938709] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4818.416893] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4823.812099] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4830.335199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4830.635533] Lustre: lustre-OST0000: new disk, initializing [ 4830.640801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4830.646849] Lustre: 168045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4830.718471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4836.244169] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4836.950202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4836.963277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4837.059231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4845.149607] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.430991] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4850.600901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4854.838271] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4855.088150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4857.829669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4857.832743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.836425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4857.840944] LustreError: Skipped 1 previous similar message [ 4857.866187] Lustre: Skipped 1 previous similar message [ 4868.069617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.079182] Lustre: Skipped 4 previous similar messages [ 4871.137154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4871.339368] Lustre: server umount lustre-OST0000 complete [ 4876.257809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.271257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.473428] Lustre: server umount lustre-MDT0000 complete [ 4883.428479] LustreError: 166087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.452722] LustreError: 166087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4885.872059] LustreError: 166068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873192 with bad export cookie 4004225059975761040 [ 4885.882777] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.894355] LustreError: 166068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4886.460436] Lustre: server umount lustre-MDT0001 complete [ 4895.981461] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:26:40 (1787873200) [ 4911.818752] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 4922.438178] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 4957.829910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.075539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4958.110584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4958.211859] Lustre: lustre-MDT0000: new disk, initializing [ 4958.338762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.362221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4962.729185] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4974.176772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.351401] Lustre: 172801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4974.626985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4980.120947] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 4989.739084] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4995.719676] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5003.700903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.040326] Lustre: lustre-OST0000: new disk, initializing [ 5004.044197] Lustre: Skipped 1 previous similar message [ 5004.048061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5004.054686] Lustre: Skipped 2 previous similar messages [ 5004.064141] Lustre: 174778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5010.895309] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5014.088391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5014.099733] Lustre: Skipped 1 previous similar message [ 5014.102276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5014.170316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5021.113504] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5029.859292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.872337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5029.872957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5029.872962] Lustre: Skipped 3 previous similar messages [ 5029.873754] Lustre: Skipped 2 previous similar messages [ 5029.893948] LustreError: Skipped 1 previous similar message [ 5042.144241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5042.308580] Lustre: server umount lustre-OST0000 complete [ 5046.242658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5046.250358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.967667] Lustre: server umount lustre-MDT0000 complete [ 5055.457695] LustreError: 172808:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.475124] LustreError: 172808:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5055.675886] LustreError: 174777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873362 with bad export cookie 4004225059975761915 [ 5055.679299] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5055.689285] LustreError: 174777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5055.987269] Lustre: server umount lustre-MDT0001 complete [ 5064.476765] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:29:29 (1787873369) [ 5080.837243] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 5091.080730] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5126.022764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.357158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5126.421573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5126.551783] Lustre: lustre-MDT0000: new disk, initializing [ 5126.659237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.677411] Lustre: Skipped 2 previous similar messages [ 5126.701864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5131.343436] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5142.191216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.332666] Lustre: 179104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5142.737812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.551685] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5157.367611] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5163.409137] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.909522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.185153] Lustre: 181079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5176.939923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5177.142555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5178.195258] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5187.816160] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5197.804417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.817525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.819055] LustreError: Skipped 1 previous similar message [ 5197.819868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.819874] Lustre: Skipped 8 previous similar messages [ 5197.871100] Lustre: Skipped 2 previous similar messages [ 5201.760431] Lustre: server umount lustre-OST0000 complete [ 5211.991958] Lustre: server umount lustre-MDT0000 complete [ 5213.156860] LustreError: 179110:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.173896] LustreError: 179110:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5215.573697] LustreError: 181089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873522 with bad export cookie 4004225059975763014 [ 5215.579929] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5215.585592] LustreError: 181089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5215.897359] Lustre: server umount lustre-MDT0001 complete [ 5226.453980] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5229.305614] Key type lgssc unregistered [ 5229.595738] LNet: 182937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5229.609393] LNetError: 182937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5229.638964] LNet: Removed LNI 192.168.206.136@tcp [ 5230.550767] Key type .llcrypt unregistered [ 5230.554091] Key type ._llcrypt unregistered [ 5254.297059] Key type ._llcrypt registered [ 5254.298900] Key type .llcrypt registered [ 5254.391327] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5255.722190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5255.747130] alg: No test for adler32 (adler32-zlib) [ 5256.855650] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 5257.176721] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5258.848220] Key type lgssc registered [ 5260.043853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5269.149248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5269.186481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.950603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5275.035195] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5281.253792] LustreError: 184354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.268625] LustreError: 184354:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5282.877520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.424973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.549175] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5294.015788] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5299.703196] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5305.378317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.837828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.515877] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5317.140253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5320.213932] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5339.620719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5339.630102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.651116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5342.691595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.693587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.217715] Lustre: server umount lustre-OST0000 complete [ 5347.810521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.834331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.850621] Lustre: Skipped 2 previous similar messages [ 5352.933193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.946583] Lustre: Skipped 1 previous similar message [ 5353.789061] Lustre: server umount lustre-MDT0000 complete [ 5357.737531] LustreError: 184333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873664 with bad export cookie 6964494362686954803 [ 5357.748994] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.754340] LustreError: 184333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5358.155710] Lustre: server umount lustre-MDT0001 complete [ 5371.715932] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5375.092452] Key type lgssc unregistered [ 5375.451349] LNet: 187910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5375.461697] LNetError: 187910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5375.484630] LNet: Removed LNI 192.168.206.136@tcp [ 5376.615734] Key type .llcrypt unregistered [ 5376.617532] Key type ._llcrypt unregistered [ 5401.487550] Key type ._llcrypt registered [ 5401.489015] Key type .llcrypt registered [ 5401.578736] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5402.206519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.290552] alg: No test for adler32 (adler32-zlib) [ 5403.312074] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 5403.528415] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5405.240478] Key type lgssc registered [ 5406.306946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5414.326697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5414.363661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.910022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.449982] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5426.148551] LustreError: 189330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.197519] LustreError: 189330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5431.273746] LustreError: 189331:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5431.990622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.284823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5436.015279] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5443.810837] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5448.927530] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5455.175783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5455.455607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.613848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5461.731590] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5468.620566] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5476.322704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5476.336761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.348107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5480.847738] Lustre: server umount lustre-OST0000 complete [ 5488.247404] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e4d879ca000.ost_server_uuid 50 [ 5489.708174] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5503.969612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5503.980607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.994155] Lustre: Skipped 1 previous similar message [ 5503.999129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.005123] Lustre: Skipped 2 previous similar messages [ 5507.042399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.042760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.398481] Lustre: server umount lustre-MDT0000 complete [ 5512.161690] LustreError: 189326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.185899] LustreError: 189326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5512.592602] LustreError: 189312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873819 with bad export cookie 69638543425058969 [ 5512.596506] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5512.602278] LustreError: 189312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5513.005577] Lustre: server umount lustre-MDT0001 complete [ 5522.819229] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:37:07 (1787873827) [ 5529.727317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.241740] LustreError: 193245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.309568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.663307] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5535.725406] LustreError: 193246:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.974458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.325761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.762122] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5553.570343] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5559.407178] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5566.322549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.837358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.427947] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5576.061981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5581.763713] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5587.900846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5588.101749] Lustre: lustre-OST0001: new disk, initializing [ 5588.104728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5588.111780] Lustre: 196084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5588.209720] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5594.365466] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5595.737771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5595.748908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5595.835811] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5603.393851] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5606.379551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.388957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.392944] Lustre: Skipped 1 previous similar message [ 5606.397396] Lustre: Skipped 1 previous similar message [ 5611.509130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5611.516701] Lustre: Skipped 3 previous similar messages [ 5619.532463] Lustre: lustre-OST0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 5619.541951] Lustre: Skipped 3 previous similar messages [ 5620.193586] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5620.330630] Lustre: server umount lustre-OST0000 complete [ 5621.738912] LustreError: 195065:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.761055] LustreError: 195065:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5628.062939] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5628.359392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5632.424871] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5632.536625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5640.010925] LustreError: 195066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.030794] LustreError: 195066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5647.329232] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.329742] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.336640] LustreError: Skipped 1 previous similar message [ 5647.343085] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5647.363816] Lustre: Skipped 1 previous similar message [ 5652.392697] Lustre: server umount lustre-OST0001 complete [ 5657.571396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.476974] Lustre: server umount lustre-MDT0000 complete [ 5666.646862] LustreError: 193226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873973 with bad export cookie 69638543425060040 [ 5666.649917] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.656749] LustreError: 193226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.965841] Lustre: server umount lustre-MDT0001 complete [ 5692.517292] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:39:57 (1787873997) [ 5700.878352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.145369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5701.186610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5701.415554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.849470] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5715.488607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.591490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5715.642872] Lustre: 199523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5719.760544] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5727.417815] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.541268] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5741.602605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5741.804567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5747.797247] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5750.982283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5754.386329] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5761.302487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.489032] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5768.419977] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5776.258327] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5781.479683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5781.480574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.486726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5781.486735] Lustre: Skipped 3 previous similar messages [ 5781.505852] Lustre: Skipped 2 previous similar messages [ 5785.291664] Lustre: server umount lustre-OST0000 complete [ 5786.607107] LustreError: 201954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.633501] LustreError: 201954:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5801.446510] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.452592] Lustre: server umount lustre-OST0001 complete [ 5817.642872] Lustre: server umount lustre-MDT0000 complete [ 5821.107307] LustreError: 200271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874127 with bad export cookie 69638543425061440 [ 5821.114294] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5821.118868] LustreError: 200271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5821.413289] Lustre: server umount lustre-MDT0001 complete [ 5844.125854] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:42:29 (1787874149) [ 5858.817795] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 5867.276209] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5898.792247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.003514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5899.024357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5899.132710] Lustre: lustre-MDT0000: new disk, initializing [ 5899.311029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.322649] Lustre: Skipped 3 previous similar messages [ 5899.372047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5903.705305] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5912.925968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.014825] Lustre: 207625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5913.056878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5913.061681] Lustre: Skipped 1 previous similar message [ 5913.130796] Lustre: lustre-MDT0001: new disk, initializing [ 5913.229071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5913.239102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5916.058094] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5923.205201] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5927.632984] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.601143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.887955] Lustre: lustre-OST0000: new disk, initializing [ 5933.898923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5933.913746] Lustre: 209593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5939.633080] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5941.839338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5941.862089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5941.967605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5948.770920] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5952.708905] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5952.873659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5956.350985] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5956.541820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5962.723623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.724816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.731088] LustreError: Skipped 2 previous similar messages [ 5962.736396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.749045] Lustre: Skipped 5 previous similar messages [ 5962.750319] Lustre: Skipped 10 previous similar messages [ 5964.752947] Lustre: server umount lustre-OST0000 complete [ 5974.819655] Lustre: server umount lustre-MDT0000 complete [ 5978.079237] LustreError: 207619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874284 with bad export cookie 69638543425063008 [ 5978.083660] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5978.101619] LustreError: 207619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5978.366510] Lustre: server umount lustre-MDT0001 complete [ 5987.089592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.712138] LustreError: 211809:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.736505] LustreError: 211809:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 5992.334246] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6001.082234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.256667] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6013.703257] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.938066] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6024.440347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.125542] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6037.794278] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.373700] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6042.554795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6047.060313] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6047.261357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6052.323160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6052.335571] LustreError: Skipped 1 previous similar message [ 6052.345750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.361653] Lustre: Skipped 2 previous similar messages [ 6055.401607] Lustre: server umount lustre-OST0000 complete [ 6061.371110] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6065.173813] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6097.455473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.009592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.017607] Lustre: Skipped 5 previous similar messages [ 6104.113854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.132211] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6104.662895] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6112.152510] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6113.321529] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6118.693625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6118.696761] Lustre: Skipped 6 previous similar messages [ 6122.884238] Lustre: server umount lustre-OST0000 complete [ 6132.811364] Lustre: server umount lustre-MDT0000 complete [ 6134.243626] LustreError: 211810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.261509] LustreError: 211810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6136.198085] LustreError: 211789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874442 with bad export cookie 69638543425063883 [ 6136.202127] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.206883] LustreError: 211789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6136.539767] Lustre: server umount lustre-MDT0001 complete [ 6144.968776] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:47:29 (1787874449) [ 6152.720311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.970177] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6167.852411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.362289] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6180.252495] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6185.388862] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.629142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.518944] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6206.123650] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6210.284421] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6210.457399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6214.538428] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6214.729420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6221.101749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6221.275245] Lustre: lustre-OST0001: new disk, initializing [ 6221.278367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6221.290166] Lustre: 221144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6226.805201] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6227.593694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6227.614282] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6227.755298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6238.252329] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.107257] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6243.365939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6248.327296] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6248.578605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6253.539652] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6253.545868] LustreError: Skipped 1 previous similar message [ 6253.549724] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.565608] Lustre: Skipped 5 previous similar messages [ 6256.647907] Lustre: server umount lustre-OST0001 complete [ 6262.984678] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6263.170941] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6267.892443] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6268.072682] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6297.730291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.219526] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6303.240259] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6303.240984] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6303.268782] Lustre: Skipped 1 previous similar message [ 6303.623548] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6310.512591] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6312.179345] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6318.574485] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6318.586334] LustreError: Skipped 2 previous similar messages [ 6320.464669] Lustre: server umount lustre-OST0001 complete [ 6342.589812] Lustre: server umount lustre-OST0000 complete [ 6360.033593] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6360.146607] Lustre: server umount lustre-MDT0000 complete [ 6363.037742] LustreError: 217759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874669 with bad export cookie 69638543425064849 [ 6363.041058] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.049428] LustreError: 217759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6369.566190] Lustre: server umount lustre-MDT0001 complete [ 6393.210313] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:51:37 (1787874697) [ 6400.268543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.575641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6400.594527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6400.752360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.756069] Lustre: Skipped 5 previous similar messages [ 6404.861546] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6412.355610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.439623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6412.464143] Lustre: 226400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6412.520038] LustreError: 226412:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.530788] LustreError: 226412:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 33 previous similar messages [ 6416.606467] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6423.750692] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6429.718866] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6436.652688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6436.959679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6443.487392] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6451.255578] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.442598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.477514] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6466.934131] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6474.849125] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.378555] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6480.932587] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6481.106545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6486.443937] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6486.718600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6492.459127] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4d8824e000.ost_server_uuid 50 [ 6494.119797] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4d8824e000.ost_server_uuid in IDLE state after 0 sec [ 6495.212829] Lustre: Permanently deactivating lustre-OST0001 [ 6495.233329] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6499.215397] Lustre: Permanently reactivating lustre-OST0001 [ 6499.239113] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6499.251264] Lustre: Skipped 2 previous similar messages [ 6504.424438] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6504.437728] LustreError: Skipped 3 previous similar messages [ 6504.443462] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6504.447504] Lustre: Skipped 20 previous similar messages [ 6518.243559] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6518.379830] Lustre: server umount lustre-OST0001 complete [ 6523.882541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.905593] Lustre: Skipped 9 previous similar messages [ 6528.501409] Lustre: server umount lustre-OST0000 complete [ 6539.110962] Lustre: server umount lustre-MDT0000 complete [ 6542.956918] LustreError: 226393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874849 with bad export cookie 69638543425066137 [ 6542.966758] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.975637] LustreError: 226393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6543.485680] Lustre: server umount lustre-MDT0001 complete [ 6569.112064] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:54:33 (1787874873) [ 6577.169050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6577.354436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6577.372870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6577.388790] Lustre: Skipped 2 previous similar messages [ 6581.597722] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6591.565082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.013575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6592.114878] Lustre: 233101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.157957] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6607.570814] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.916520] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6620.980212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.117328] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6621.137825] Lustre: Permanently deactivating lustre-OST0000 [ 6621.146330] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6628.032891] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6629.968399] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6636.853844] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6645.840816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.904595] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6653.422310] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6655.685401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6660.911094] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.561253] Lustre: Permanently reactivating lustre-OST0000 [ 6662.576781] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6662.581750] Lustre: Skipped 2 previous similar messages [ 6669.846678] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6669.854510] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 000000001950c846, cur 1787874976 deadline 1787874966 last 1787874936 [ 6669.902068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6685.818514] Lustre: server umount lustre-OST0001 complete [ 6704.096474] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6704.321280] Lustre: server umount lustre-OST0000 complete [ 6714.487398] Lustre: server umount lustre-MDT0000 complete [ 6718.845154] LustreError: 233093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875025 with bad export cookie 69638543425067782 [ 6718.853210] LustreError: 233093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6718.855090] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6719.276690] Lustre: server umount lustre-MDT0001 complete [ 6732.160750] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6735.984815] Key type lgssc unregistered [ 6736.313734] LNet: 237993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6736.327386] LNetError: 237993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6736.348736] LNet: Removed LNI 192.168.206.136@tcp [ 6737.477926] Key type .llcrypt unregistered [ 6737.495245] Key type ._llcrypt unregistered [ 6762.259339] Key type ._llcrypt registered [ 6762.261892] Key type .llcrypt registered [ 6762.427220] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6763.359857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6763.416620] alg: No test for adler32 (adler32-zlib) [ 6764.419124] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 6764.653995] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6766.272364] Key type lgssc registered [ 6767.356680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6777.678186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6777.701915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.250075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.950319] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6788.585723] LustreError: 239461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6788.601736] LustreError: 239461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6793.138306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.328161] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6793.373595] Lustre: 239448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6793.392820] Lustre: Permanently deactivating lustre-MDT0001 [ 6793.431343] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6793.706735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6793.885270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6799.615262] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6800.433548] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6808.598314] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6814.953526] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6822.045185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.448268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.487035] LustreError: 241302:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.501634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6828.338893] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6828.520298] LustreError: 241304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.635540] LustreError: 241360:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.704447] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6838.764846] LustreError: 241305:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.776484] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6838.800943] LustreError: 241305:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6845.203556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.603537] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6851.095281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6851.099293] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6854.357151] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6863.533827] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.654820] Lustre: Permanently reactivating lustre-MDT0001 [ 6866.675469] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6866.680175] Lustre: Skipped 2 previous similar messages [ 6872.419667] Lustre: lustre-MDT0001: Client eae9a0ce-f84d-4b66-9fea-c8a6b72ced45 (at 192.168.206.36@tcp) reconnecting [ 6873.717811] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.732311] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6873.739475] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6873.761095] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6878.726785] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6880.252262] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6886.732665] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6886.949639] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6891.125649] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6891.414719] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6894.214368] Lustre: Permanently deactivating lustre-MDT0001 [ 6894.264214] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6894.274735] Lustre: Skipped 2 previous similar messages [ 6899.788027] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6909.377855] Lustre: server umount lustre-MDT0000 complete [ 6909.415543] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.417821] LustreError: 239462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.422762] Lustre: Skipped 3 previous similar messages [ 6909.432130] LustreError: 239462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6913.974747] LustreError: 239442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875220 with bad export cookie 18040858708473929988 [ 6914.002276] LustreError: 239442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6914.013156] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6914.331092] Lustre: server umount lustre-MDT0001 complete [ 6924.207320] Lustre: server umount lustre-OST0000 complete [ 6933.864333] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:00:38 (1787875238) [ 6943.308972] Lustre: server umount lustre-OST0001 complete [ 6950.894184] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 6968.398463] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7023.648621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.015674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7024.026967] Lustre: Skipped 2 previous similar messages [ 7024.084673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7024.212505] Lustre: lustre-MDT0000: new disk, initializing [ 7024.343626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.393328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7029.826910] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7041.758253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.879826] Lustre: 247764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7041.927365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7041.929809] Lustre: Skipped 1 previous similar message [ 7041.942815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7042.016387] Lustre: lustre-MDT0001: new disk, initializing [ 7042.119884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7042.176125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7042.189785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7047.195132] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7056.446516] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7062.168923] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7068.240251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.475223] Lustre: lustre-OST0000: new disk, initializing [ 7068.479084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7068.485532] Lustre: 249733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7068.565672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.546124] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7075.900808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7075.908474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7076.010461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7082.847230] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7084.885751] LustreError: 250028:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7093.088700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.296784] Lustre: lustre-OST0001: new disk, initializing [ 7093.305421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7093.319182] Lustre: 250851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7094.974174] LustreError: 250028:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7095.010421] LustreError: 247772:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7098.744429] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7098.931794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7098.936941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7099.075570] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7105.058642] LustreError: 247772:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7107.482084] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.811451] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7112.030191] Lustre: lustre-OST0001: Not available for connect from 192.168.206.36@tcp (stopping) [ 7114.732402] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7114.733888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.740558] LustreError: Skipped 1 previous similar message [ 7114.744888] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7114.753229] Lustre: Skipped 1 previous similar message [ 7114.778229] Lustre: Skipped 1 previous similar message [ 7116.621223] Lustre: server umount lustre-OST0001 complete [ 7117.130873] LustreError: 249717:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.215203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.233324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7135.239690] Lustre: Skipped 1 previous similar message [ 7139.133557] Lustre: server umount lustre-OST0000 complete [ 7144.420534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7144.430456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.450688] Lustre: Skipped 1 previous similar message [ 7144.456947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.679531] Lustre: server umount lustre-MDT0000 complete [ 7150.562650] LustreError: 249189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7150.581510] LustreError: 249189:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 7152.335522] LustreError: 247756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875459 with bad export cookie 18040858708473932802 [ 7152.348531] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.353804] LustreError: 247756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7158.710889] Lustre: server umount lustre-MDT0001 complete [ 7170.162927] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7173.176615] Key type lgssc unregistered [ 7173.471698] LNet: 252886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7173.483556] LNetError: 252886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7174.510046] LNet: Removed LNI 192.168.206.136@tcp [ 7175.613889] Key type .llcrypt unregistered [ 7175.615731] Key type ._llcrypt unregistered [ 7213.842959] Key type ._llcrypt registered [ 7213.845777] Key type .llcrypt registered [ 7213.932212] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7215.157978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7215.171512] alg: No test for adler32 (adler32-zlib) [ 7216.301602] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7216.499176] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7218.128162] Key type lgssc registered [ 7219.205372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.130038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7227.191612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.556987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7228.577431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7228.729321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.063529] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7242.870278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.968798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7243.030627] Lustre: 255096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.366814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7247.916260] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7254.727624] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7260.683635] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7267.870441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.016863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7268.212276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.162809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7274.593242] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7282.049208] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7335.393740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.398600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.405442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7335.907852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.910899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.916763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7337.546946] Lustre: server umount lustre-OST0000 complete [ 7340.245264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7348.361232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.676646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7354.513228] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7355.891738] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7355.905596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7355.924106] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7355.936844] Lustre: Skipped 1 previous similar message [ 7361.493140] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.953142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7401.957118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.967960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7401.978575] Lustre: Skipped 1 previous similar message [ 7405.921588] Lustre: server umount lustre-OST0000 complete [ 7412.195639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7412.197745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.208586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.208593] Lustre: Skipped 1 previous similar message [ 7412.240355] Lustre: Skipped 1 previous similar message [ 7415.359917] Lustre: server umount lustre-MDT0000 complete [ 7418.720520] LustreError: 255086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875725 with bad export cookie 3821654577515565482 [ 7418.725613] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7418.731580] LustreError: 255086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7419.000537] Lustre: server umount lustre-MDT0001 complete [ 7428.282322] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7430.815227] Key type lgssc unregistered [ 7431.091367] LNet: 260520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.098647] LNetError: 260520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.114696] LNet: Removed LNI 192.168.206.136@tcp [ 7431.957287] Key type .llcrypt unregistered [ 7431.963758] Key type ._llcrypt unregistered [ 7454.833216] Key type ._llcrypt registered [ 7454.835696] Key type .llcrypt registered [ 7454.936630] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7455.603571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7455.702856] alg: No test for adler32 (adler32-zlib) [ 7456.664049] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7456.846582] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7458.488141] Key type lgssc registered [ 7459.462310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7465.828248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7465.844901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.250786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.066638] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7476.708110] LustreError: 261938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.721063] LustreError: 261938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7477.847552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.122479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.837836] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7486.908598] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.287746] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.568266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.961827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.019520] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7509.434923] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7511.286363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7537.122837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7537.140455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.167099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.170612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.184276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.197449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.219400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.235049] Lustre: Skipped 1 previous similar message [ 7542.476855] Lustre: server umount lustre-OST0000 complete [ 7549.924219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7549.941506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7549.961559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7553.252770] Lustre: server umount lustre-MDT0000 complete [ 7556.578173] LustreError: 261938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.611168] LustreError: 261938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7557.750882] LustreError: 261920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875864 with bad export cookie 12234010387862121511 [ 7557.758491] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7557.768344] LustreError: 261920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7558.378193] Lustre: server umount lustre-MDT0001 complete [ 7574.866411] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7578.102283] Key type lgssc unregistered [ 7578.458835] LNet: 266248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7578.477924] LNetError: 266248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7579.503555] LNet: Removed LNI 192.168.206.136@tcp [ 7580.751319] Key type .llcrypt unregistered [ 7580.754441] Key type ._llcrypt unregistered [ 7598.749921] Key type ._llcrypt registered [ 7598.751800] Key type .llcrypt registered [ 7598.850732] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7600.991497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7601.547699] alg: No test for adler32 (adler32-zlib) [ 7602.619603] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7602.935050] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7604.672912] Key type lgssc registered [ 7606.257534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7618.861497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7618.882251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.445866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.753851] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7630.834122] LustreError: 267471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.855889] LustreError: 267471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7635.938631] LustreError: 267472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.117792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.623155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7643.005461] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7650.738793] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7657.082126] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7667.012519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.840964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.160841] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7686.425114] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7690.366177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7704.887067] Lustre: server umount lustre-OST0000 complete [ 7710.177242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7710.191387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.215601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.202932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.204734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.240123] Lustre: Skipped 1 previous similar message [ 7716.326342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.345196] Lustre: Skipped 1 previous similar message [ 7718.485995] Lustre: server umount lustre-MDT0000 complete [ 7720.352682] Lustre: 266780:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876011/real 1787876011] req@0000000077689186 x1874723769187584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787876027 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7720.387628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.400769] LustreError: 268199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.439243] LustreError: 268199:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7721.440617] Lustre: 266780:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876012/real 1787876012] req@00000000ed202742 x1874723769188608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787876028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7722.528685] LustreError: 267452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876029 with bad export cookie 16163672080512061164 [ 7722.530113] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.539836] LustreError: 267452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7723.184410] Lustre: server umount lustre-MDT0001 complete [ 7736.015800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7738.652414] Key type lgssc unregistered [ 7739.036987] LNet: 271157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7739.043593] LNetError: 271157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7739.060595] LNet: Removed LNI 192.168.206.136@tcp [ 7740.250635] Key type .llcrypt unregistered [ 7740.253352] Key type ._llcrypt unregistered [ 7768.647905] Key type ._llcrypt registered [ 7768.653126] Key type .llcrypt registered [ 7768.783316] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7769.732472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7769.907420] alg: No test for adler32 (adler32-zlib) [ 7770.982902] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7771.198860] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7772.867476] Key type lgssc registered [ 7774.171580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.111633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7781.138655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7782.582300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.618987] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7793.122615] LustreError: 272577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.133464] LustreError: 272577:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7794.164764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.515919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7799.288174] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7807.059043] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7813.399883] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7821.022651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7821.441405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.599073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7827.691360] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7834.410637] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7861.735219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.736041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.758538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7861.766179] Lustre: Skipped 1 previous similar message [ 7866.852750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.858400] Lustre: Skipped 1 previous similar message [ 7867.480779] Lustre: server umount lustre-OST0000 complete [ 7871.459115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.469574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.497082] Lustre: Skipped 1 previous similar message [ 7871.513338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.095678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.108310] Lustre: Skipped 2 previous similar messages [ 7877.800266] Lustre: server umount lustre-MDT0000 complete [ 7882.210339] LustreError: 272578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.217693] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876188 with bad export cookie 2418728736007111896 [ 7882.220507] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.222330] LustreError: 272578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7882.259842] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7882.484324] Lustre: server umount lustre-MDT0001 complete [ 7892.510823] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7895.076774] Key type lgssc unregistered [ 7895.326824] LNet: 276884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7895.334559] LNetError: 276884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7895.359498] LNet: Removed LNI 192.168.206.136@tcp [ 7896.154858] Key type .llcrypt unregistered [ 7896.157941] Key type ._llcrypt unregistered [ 7914.410722] Key type ._llcrypt registered [ 7914.417027] Key type .llcrypt registered [ 7914.555226] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7915.900659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7915.925792] alg: No test for adler32 (adler32-zlib) [ 7917.023495] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7917.341922] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7919.176180] Key type lgssc registered [ 7920.328738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7928.293595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7928.313210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.822523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.784114] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7940.072643] LustreError: 278121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.095767] LustreError: 278121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7942.753745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.139396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7946.934364] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7954.553739] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7960.820673] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.787859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7970.338686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7974.452430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7977.414430] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7985.253922] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8003.041848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.049663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.058050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8003.073813] Lustre: Skipped 1 previous similar message [ 8008.180289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.185320] Lustre: Skipped 2 previous similar messages [ 8008.474348] Lustre: server umount lustre-OST0000 complete [ 8013.282458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8013.308879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8013.320146] Lustre: Skipped 1 previous similar message [ 8018.401502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8018.407652] Lustre: Skipped 1 previous similar message [ 8023.524973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8023.534333] Lustre: Skipped 1 previous similar message [ 8027.104137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8027.347794] Lustre: server umount lustre-MDT0000 complete [ 8028.641676] LustreError: 278113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.661471] LustreError: 278113:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8030.871758] LustreError: 278087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876337 with bad export cookie 693473900188743836 [ 8030.885169] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8031.185520] Lustre: server umount lustre-MDT0001 complete [ 8041.679930] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 8044.088685] Key type lgssc unregistered [ 8044.363559] LNet: 281827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.373066] LNetError: 281827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.392987] LNet: Removed LNI 192.168.206.136@tcp [ 8045.100662] Key type .llcrypt unregistered [ 8045.103167] Key type ._llcrypt unregistered [ 8057.678633] Key type ._llcrypt registered [ 8057.681196] Key type .llcrypt registered [ 8057.786618] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8073.642066] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8090.540498] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8091.875346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8091.901654] alg: No test for adler32 (adler32-zlib) [ 8092.989521] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8093.276515] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 8094.985096] Key type lgssc registered [ 8097.105580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8138.613702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8138.635985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.921712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8139.996167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8140.178334] Lustre: lustre-MDT0000: new disk, initializing [ 8140.386563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.427447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8144.691211] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8155.710332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.856804] Lustre: 285389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8155.949840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8155.961128] Lustre: Skipped 1 previous similar message [ 8156.048788] Lustre: lustre-MDT0001: new disk, initializing [ 8156.160292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8156.192823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8156.200670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8161.020120] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8170.048582] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8176.216390] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8182.813380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.002248] Lustre: lustre-OST0000: new disk, initializing [ 8183.006436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8183.013676] Lustre: 287357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8183.075806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.306095] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8192.606109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8192.628849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8192.760078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8198.952133] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8202.902584] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8203.062913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8207.407648] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8207.549374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8213.486168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.487864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8213.493437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.495628] Lustre: Skipped 1 previous similar message [ 8213.532980] LustreError: Skipped 1 previous similar message [ 8216.131800] Lustre: server umount lustre-OST0000 complete [ 8222.693464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8222.722623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8222.754295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.757320] Lustre: Skipped 1 previous similar message [ 8223.729931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.379704] Lustre: server umount lustre-MDT0000 complete [ 8232.441792] LustreError: 285380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876539 with bad export cookie 7753505017179427797 [ 8232.443514] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8232.473797] LustreError: 285380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8232.790647] Lustre: server umount lustre-MDT0001 complete [ 8243.532540] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:22:27 (1787876547) [ 8249.972499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.374304] LustreError: 289769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.395029] LustreError: 289769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8250.463893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.910512] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8255.478639] LustreError: 289770:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.580599] LustreError: 289769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8262.977933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.366362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8267.691988] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8276.251581] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.422932] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8288.755044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8289.117753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.420302] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8302.571725] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8315.877259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.891712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.912356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.920845] Lustre: Skipped 2 previous similar messages [ 8318.303209] Lustre: server umount lustre-OST0000 complete [ 8325.092736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.105040] LustreError: Skipped 1 previous similar message [ 8325.111805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.138691] Lustre: Skipped 1 previous similar message [ 8325.146512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.154826] Lustre: Skipped 2 previous similar messages [ 8328.538372] Lustre: server umount lustre-MDT0000 complete [ 8332.257456] LustreError: 290498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.282417] LustreError: 290498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8332.728400] LustreError: 289749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876639 with bad export cookie 7753505017179428679 [ 8332.735366] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.742896] LustreError: 289749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8333.219868] Lustre: server umount lustre-MDT0001 complete [ 8345.294069] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 8348.641478] Key type lgssc unregistered [ 8349.037579] LNet: 293312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8349.061573] LNetError: 293312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8349.088195] LNet: Removed LNI 192.168.206.136@tcp [ 8349.974304] Key type .llcrypt unregistered [ 8349.977428] Key type ._llcrypt unregistered [ 8378.342193] Key type ._llcrypt registered [ 8378.347459] Key type .llcrypt registered [ 8378.425684] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8392.297779] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8393.451090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.504845] alg: No test for adler32 (adler32-zlib) [ 8394.519869] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8394.797107] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 8396.496782] Key type lgssc registered [ 8397.474042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8439.237734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8439.257033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.511443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8440.575486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8440.631919] Lustre: lustre-MDT0000: new disk, initializing [ 8440.718813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.752395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8444.957691] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8454.154483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.338398] Lustre: 297143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8454.447395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8454.459159] Lustre: Skipped 1 previous similar message [ 8454.560750] Lustre: lustre-MDT0001: new disk, initializing [ 8454.691962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8454.728140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8454.735533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8459.225105] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8469.282416] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.116929] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8481.005586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.297975] Lustre: lustre-OST2710: new disk, initializing [ 8481.300704] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8481.305247] Lustre: 299109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8481.382569] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8487.086459] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8490.059885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8490.073654] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8490.263495] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8496.488344] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.920725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8504.089403] Lustre: lustre-OST03e8: new disk, initializing [ 8504.096233] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8504.106956] Lustre: 300172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8504.236458] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8509.865267] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8512.099713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8512.116311] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8512.164033] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8518.340515] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8524.361078] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8524.590213] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8528.486060] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8528.678898] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8533.371726] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8533.617477] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8538.156549] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8538.322826] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8546.786277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8546.807667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.820046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.330304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.334028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.399875] Lustre: server umount lustre-MDT0000 complete [ 8553.446188] LustreError: 297154:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.468787] LustreError: 297154:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 8554.151345] LustreError: 300182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876860 with bad export cookie 16195362629626310630 [ 8554.153192] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8554.157978] LustreError: 300182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8554.406434] Lustre: server umount lustre-MDT0001 complete [ 8567.867560] Lustre: server umount lustre-OST2710 complete [ 8581.955301] Lustre: server umount lustre-OST03e8 complete [ 8597.370870] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8607.383665] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8648.562348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.854597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8648.879510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8648.969354] Lustre: lustre-MDT0000: new disk, initializing [ 8649.083694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8649.111700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8653.601109] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8669.260317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.402472] Lustre: 305427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8669.502259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8669.510755] Lustre: Skipped 1 previous similar message [ 8669.598735] Lustre: lustre-MDT0001: new disk, initializing [ 8669.691296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8669.736047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8669.745447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8674.867184] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8684.991230] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8690.601510] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8697.162692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8697.462181] Lustre: lustre-OST0000: new disk, initializing [ 8697.466954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8697.471931] Lustre: 307393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8697.561823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.058817] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8706.693792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8706.708623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8706.863719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8716.375979] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8722.335220] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8722.589509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8727.982688] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8728.287509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8732.647117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.648051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8732.653527] Lustre: Skipped 2 previous similar messages [ 8732.659876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8732.671385] Lustre: Skipped 4 previous similar messages [ 8736.123290] Lustre: server umount lustre-OST0000 complete [ 8741.355053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8741.361295] LustreError: Skipped 1 previous similar message [ 8741.369472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.392524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8746.323826] Lustre: server umount lustre-MDT0000 complete [ 8748.001301] LustreError: 305438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.018880] LustreError: 305438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8751.010872] LustreError: 305434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.029099] LustreError: 305434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8751.362901] LustreError: 305419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877058 with bad export cookie 16195362629626312324 [ 8751.423667] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8751.966598] Lustre: server umount lustre-MDT0001 complete [ 8768.434045] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:31:12 (1787877072) [ 8770.942561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8772.714825] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:31:17 (1787877077) [ 8775.977686] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8806.111153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8806.401875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8806.428795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8806.615867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8811.586065] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8820.762065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.892358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8820.953321] Lustre: 311254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.729400] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8849.326109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8849.622651] Lustre: Denying initial registration attempt from nid 192.168.206.136@tcp, specified as failover [ 8849.635635] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8849.643713] LustreError: 312711:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8849.649674] LustreError: 312711:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8849.659548] LustreError: 312711:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8849.745647] Lustre: server umount lustre-OST0000 complete [ 8849.750570] LustreError: 312711:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8857.065189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8857.070535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.084559] Lustre: Skipped 1 previous similar message [ 8857.090106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.093600] Lustre: Skipped 2 previous similar messages [ 8862.539189] Lustre: server umount lustre-MDT0000 complete [ 8864.243895] LustreError: 311261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.284698] LustreError: 311261:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8865.997467] LustreError: 311998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877172 with bad export cookie 16195362629626313206 [ 8866.009012] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8866.009720] LustreError: 311998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8866.277946] Lustre: server umount lustre-MDT0001 complete [ 8880.056837] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 8883.028795] Key type lgssc unregistered [ 8883.488415] LNet: 313833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8883.499850] LNetError: 313833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8883.528472] LNet: Removed LNI 192.168.206.136@tcp [ 8884.558946] Key type .llcrypt unregistered [ 8884.561486] Key type ._llcrypt unregistered [ 8896.720801] Key type ._llcrypt registered [ 8896.725685] Key type .llcrypt registered [ 8896.911883] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8897.803666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8897.889512] alg: No test for adler32 (adler32-zlib) [ 8898.973568] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8899.314611] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 8900.984164] Key type lgssc registered [ 8902.264593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.269822] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8954.386707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8954.399220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.608357] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8955.623825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8955.756565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.704686] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8968.164589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.282926] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8968.351307] Lustre: 316601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8968.820821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8973.822857] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8980.927706] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8986.278353] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9008.187212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.477553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9008.838376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.378190] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9025.956614] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9030.115257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9030.117223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.126740] LustreError: Skipped 1 previous similar message [ 9030.127398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9030.170793] Lustre: Skipped 1 previous similar message [ 9035.253907] Lustre: server umount lustre-OST0000 complete [ 9040.353922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.358920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.371811] Lustre: Skipped 1 previous similar message [ 9040.389398] Lustre: Skipped 2 previous similar messages [ 9045.447993] Lustre: server umount lustre-MDT0000 complete [ 9050.563174] LustreError: 316592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877357 with bad export cookie 5370003391230355563 [ 9050.568256] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9050.571200] LustreError: 316592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9051.167536] Lustre: server umount lustre-MDT0001 complete [ 9063.401511] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 9066.287623] Key type lgssc unregistered [ 9066.975732] LNet: 320176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9066.987781] LNetError: 320176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9067.033420] LNet: Removed LNI 192.168.206.136@tcp [ 9068.087929] Key type .llcrypt unregistered [ 9068.090850] Key type ._llcrypt unregistered [ 9099.306164] Key type ._llcrypt registered [ 9099.307603] Key type .llcrypt registered [ 9099.489801] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9100.433730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9100.452149] alg: No test for adler32 (adler32-zlib) [ 9101.414778] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 9101.653896] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 9103.354083] Key type lgssc registered [ 9104.690901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9114.855596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9114.872829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.633443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9123.277471] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9126.887501] LustreError: 321608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.915327] LustreError: 321608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9132.005212] LustreError: 321607:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9135.768648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.249371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9143.246821] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9152.892621] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9159.367968] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9168.106710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9168.423329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9174.929381] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9182.724932] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9197.027543] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9197.033468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.048321] Lustre: Skipped 1 previous similar message [ 9197.068745] Lustre: Skipped 2 previous similar messages [ 9200.010806] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 9202.150486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9202.164887] Lustre: Skipped 2 previous similar messages [ 9205.068882] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 9210.182703] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 9210.192656] Lustre: Skipped 3 previous similar messages [ 9211.362293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9211.489381] Lustre: server umount lustre-MDT0000 complete [ 9212.391113] LustreError: 321608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.424229] LustreError: 321608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9215.305794] LustreError: 321603:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.370050] LustreError: 321587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877522 with bad export cookie 17058204363169470463 [ 9215.386728] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.514037] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.531545] Lustre: Skipped 1 previous similar message [ 9220.425860] LustreError: 322849:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.455870] LustreError: 322849:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9220.464524] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [ 9220.473914] Lustre: Skipped 1 previous similar message [ 9222.040688] Lustre: server umount lustre-MDT0001 complete [ 9229.659924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9239.663475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.864226] LustreError: 325210:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9240.873758] LustreError: 325210:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000009fc1b0c x1874725341016064/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787877547 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9240.894823] LustreError: 325210:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9241.056691] LustreError: 320917:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000093a274c8 x1874725341016448/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9241.256210] LustreError: 323441:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9241.273966] LustreError: 323441:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9241.331606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.394411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 9245.522596] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9250.535847] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9253.451119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.636110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9258.038582] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9258.988567] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9265.427376] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9267.710681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9273.769352] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9275.452208] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9282.530617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9282.537472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9282.553181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.557704] Lustre: Skipped 1 previous similar message [ 9284.580060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.328322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9295.491728] Lustre: server umount lustre-OST0000 complete [ 9299.939846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.956908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9299.968575] Lustre: Skipped 1 previous similar message [ 9313.249033] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9313.475946] Lustre: server umount lustre-MDT0000 complete [ 9315.309279] LustreError: 325228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.346122] LustreError: 325228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9317.841065] LustreError: 321589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877624 with bad export cookie 17058204363169478093 [ 9317.841921] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.856246] LustreError: 321589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9318.432627] Lustre: server umount lustre-MDT0001 complete [ 9332.890651] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 9335.678384] Key type lgssc unregistered [ 9336.106770] LNet: 327903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9336.116824] LNetError: 327903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.193459] LNet: Removed LNI 192.168.206.136@tcp [ 9338.316071] Key type .llcrypt unregistered [ 9338.318787] Key type ._llcrypt unregistered [ 9366.776497] Key type ._llcrypt registered [ 9366.783048] Key type .llcrypt registered [ 9366.968423] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9369.200187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9369.256236] alg: No test for adler32 (adler32-zlib) [ 9370.341724] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 9370.610515] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 9372.272459] Key type lgssc registered [ 9373.861325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9383.749944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9383.829513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.536990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.372267] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9395.681974] LustreError: 329321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.706881] LustreError: 329321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9400.346315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.874630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9406.089305] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9414.832770] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9421.370708] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9425.959470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.971718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9425.976601] Lustre: Skipped 1 previous similar message [ 9425.998945] Lustre: Skipped 1 previous similar message [ 9430.280545] Lustre: server umount lustre-MDT0000 complete [ 9431.023709] LustreError: 329321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9431.048577] LustreError: 329321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9434.042667] LustreError: 329300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877740 with bad export cookie 15420605477271554627 [ 9434.064740] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.782330] Lustre: server umount lustre-MDT0001 complete [ 9452.623551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.181969] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9453.329538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9453.931613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9459.624156] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9469.520069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.603796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9469.635101] Lustre: 331909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9469.817714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9469.903731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9474.631350] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9483.198978] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9488.023226] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9496.817899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9497.186639] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9497.206822] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9497.224727] LustreError: 333793:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9497.234857] LustreError: 333793:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9497.242850] LustreError: 333793:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9497.347036] Lustre: server umount lustre-OST0000 complete [ 9497.352446] LustreError: 333793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9509.624195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9509.921856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9510.283404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9516.158768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [ 9517.671465] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9525.420921] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9535.060594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9535.254626] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9535.383245] Lustre: lustre-OST0001: new disk, initializing [ 9535.400508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9535.422394] Lustre: 335334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9535.554614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9541.388168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9541.408746] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9541.644725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9544.439890] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9554.388680] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9562.081294] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9562.082206] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9562.097214] LustreError: Skipped 1 previous similar message [ 9562.103420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9562.157609] Lustre: Skipped 1 previous similar message [ 9563.519091] Lustre: server umount lustre-OST0001 complete [ 9567.208076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9567.208839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.216748] LustreError: 334259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9567.224463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9567.224471] Lustre: Skipped 1 previous similar message [ 9567.284179] Lustre: Skipped 1 previous similar message [ 9572.328747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.330140] LustreError: 334261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.331471] Lustre: Skipped 2 previous similar messages [ 9572.351086] LustreError: 334261:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9572.851580] Lustre: server umount lustre-OST0000 complete [ 9577.442490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.451436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.480870] Lustre: Skipped 1 previous similar message [ 9582.562695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9582.567807] Lustre: Skipped 3 previous similar messages [ 9583.209244] Lustre: server umount lustre-MDT0000 complete [ 9587.200429] LustreError: 331902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877893 with bad export cookie 15420605477271555180 [ 9587.203915] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9587.212245] LustreError: 331902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9587.756365] Lustre: server umount lustre-MDT0001 complete [ 9599.278799] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 9602.808251] Key type lgssc unregistered [ 9603.265711] LNet: 337299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9603.280940] LNetError: 337299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9603.311273] LNet: Removed LNI 192.168.206.136@tcp [ 9604.924811] Key type .llcrypt unregistered [ 9604.929387] Key type ._llcrypt unregistered [ 9679.546866] Key type ._llcrypt registered [ 9679.557680] Key type .llcrypt registered [ 9679.633069] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 9697.210225] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9698.638686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9698.699955] alg: No test for adler32 (adler32-zlib) [ 9699.857243] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 9700.120093] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 9701.840250] Key type lgssc registered [ 9703.043510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9743.121397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9743.143548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.391857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9744.438898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9744.537788] Lustre: lustre-MDT0000: new disk, initializing [ 9744.586921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.607258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9749.136766] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9757.833379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.934587] Lustre: 342237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9757.977314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9757.983973] Lustre: Skipped 1 previous similar message [ 9758.047350] Lustre: lustre-MDT0001: new disk, initializing [ 9758.152532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.188814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9758.211398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9761.730178] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9770.429049] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9777.022872] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.273333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.495824] Lustre: lustre-OST0000: new disk, initializing [ 9784.504572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9784.511822] Lustre: 344202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.573187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9790.608892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9790.629824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9790.779659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9791.559910] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9801.345612] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.841859] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9807.190264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9812.154990] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9812.352620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.549788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.562701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.564953] Lustre: Skipped 1 previous similar message [ 9816.565567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.596464] Lustre: Skipped 1 previous similar message [ 9820.722313] Lustre: server umount lustre-OST0000 complete [ 9824.739535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9824.744543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.765905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.790647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.793780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.817852] Lustre: Skipped 1 previous similar message [ 9830.683531] Lustre: server umount lustre-MDT0000 complete [ 9834.543987] LustreError: 342230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878141 with bad export cookie 14280133808907110407 [ 9834.549738] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9834.556911] LustreError: 342230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9834.994473] Lustre: server umount lustre-MDT0001 complete [ 9844.213599] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:49:08 (1787878148) [ 9852.352947] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:49:16 (1787878156) [ 9861.285662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.901653] LustreError: 346958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9861.920827] LustreError: 346958:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9862.010662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9867.245342] LustreError: 346959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.445169] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9871.336876] LustreError: 346958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.938583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.462519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.604659] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9887.780965] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9893.878647] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9900.630823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9901.082698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.727860] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9915.315629] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9922.255227] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9923.995674] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9925.553498] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9927.296372] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9929.288372] Lustre: Failing over lustre-MDT0000 [ 9929.566680] Lustre: server umount lustre-MDT0000 complete [ 9932.131445] LustreError: 346954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9932.773296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.794079] Lustre: Skipped 1 previous similar message [ 9932.798898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9937.234517] LustreError: 346953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9937.258167] LustreError: 346953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9947.476506] LustreError: 346953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.498080] LustreError: 346953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9949.153110] Lustre: 340461:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787878239/real 1787878239] req@00000000ee970ec2 x1874725968089728/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787878255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9949.195904] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.231506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.744719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.797701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9963.572466] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9963.867485] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9964.022223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9964.081641] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9964.138751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9973.822297] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9976.009473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9979.562708] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9981.716216] Lustre: Failing over lustre-MDT0000 [ 9982.093149] Lustre: server umount lustre-MDT0000 complete [ 9984.373039] LustreError: 346954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9984.397107] LustreError: 346954:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 9984.486893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.498832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9984.501609] Lustre: Skipped 3 previous similar messages [10000.353376] Lustre: 340462:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787878291/real 1787878291] req@0000000048d0fe5e x1874725968110720/t0(0) o400->MGC192.168.206.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787878307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10000.392755] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10002.769837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.595758] LustreError: 340458:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fbefb810 x1874725968115968/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10011.145030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10011.297936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10015.065364] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10016.291196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10016.311310] Lustre: Skipped 2 previous similar messages [10016.376240] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10016.469110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10017.945128] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10029.649872] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10031.798581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10041.826112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.830283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10041.838022] Lustre: Skipped 1 previous similar message [10044.288614] Lustre: server umount lustre-MDT0000 complete [10046.953780] LustreError: 346958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.980585] LustreError: 346958:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 29 previous similar messages [10049.087223] LustreError: 346941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878355 with bad export cookie 14280133808907113130 [10049.099034] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10049.101951] LustreError: 346941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10049.405703] Lustre: server umount lustre-MDT0001 complete [10064.195158] Lustre: server umount lustre-OST0000 complete [10080.355631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10080.678252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10086.419343] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10097.234674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.904055] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10108.926813] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10113.608867] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10120.685724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.207829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10127.478530] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10135.998609] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10143.721399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10143.743469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.764679] Lustre: Skipped 1 previous similar message [10143.784249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10143.795738] Lustre: Skipped 2 previous similar messages [10148.953648] Lustre: server umount lustre-OST0000 complete [10153.954821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10153.963102] LustreError: Skipped 1 previous similar message [10153.983834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10153.995066] Lustre: Skipped 2 previous similar messages [10167.268606] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10167.585861] Lustre: server umount lustre-MDT0000 complete [10171.368740] LustreError: 354030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.422320] LustreError: 354030:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10172.415560] LustreError: 353283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878479 with bad export cookie 14280133808907113872 [10172.427618] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10172.435553] LustreError: 353283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10172.969852] Lustre: server umount lustre-MDT0001 complete [10185.838093] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10188.985441] Key type lgssc unregistered [10189.322731] LNet: 356837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10189.333723] LNetError: 356837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10189.354375] LNet: Removed LNI 192.168.206.136@tcp [10190.315303] Key type .llcrypt unregistered [10190.324528] Key type ._llcrypt unregistered [10215.768404] Key type ._llcrypt registered [10215.770569] Key type .llcrypt registered [10215.869205] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10217.096278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10217.141335] alg: No test for adler32 (adler32-zlib) [10218.326082] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [10218.626649] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10220.489099] Key type lgssc registered [10222.146821] Lustre: Echo OBD driver; http://www.lustre.org/ [10234.206619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10234.263038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.291820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10244.210530] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10245.604648] LustreError: 358359:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.629985] LustreError: 358359:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10250.721388] LustreError: 358358:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.122646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10254.742756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10260.387642] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10270.695626] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.205360] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.385336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.734893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10292.944396] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10303.207167] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10305.294307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10320.868289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.882256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.888154] Lustre: Skipped 1 previous similar message [10325.975451] Lustre: server umount lustre-MDT0000 complete [10325.990795] LustreError: 358358:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.006191] LustreError: 358358:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10329.650457] LustreError: 358340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878636 with bad export cookie 11179407203359902707 [10329.657235] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.664125] LustreError: 358340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10330.103888] Lustre: server umount lustre-MDT0001 complete [10344.526971] Lustre: server umount lustre-OST0000 complete [10358.630880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.062043] LustreError: 362094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.120119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.923968] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10364.385614] LustreError: 362095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.900690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.132055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10378.392195] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10386.836066] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10393.519470] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10403.819626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10404.424537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10411.206566] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10419.778727] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10429.045730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10444.769955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.776004] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10444.790437] Lustre: Skipped 1 previous similar message [10444.791090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10444.791094] Lustre: Skipped 2 previous similar messages [10449.907931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10449.910808] Lustre: Skipped 2 previous similar messages [10450.849385] Lustre: server umount lustre-MDT0000 complete [10454.647700] LustreError: 362075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878761 with bad export cookie 11179407203359903820 [10454.662252] LustreError: 362075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10454.663218] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10455.031305] Lustre: server umount lustre-MDT0001 complete [10469.438432] Lustre: server umount lustre-OST0000 complete [10485.787571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.377980] LustreError: 366028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.413288] LustreError: 366028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10486.501928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10491.353709] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10500.248771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.640467] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10510.744549] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10517.513654] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10525.422838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10525.739688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.750618] Lustre: Skipped 1 previous similar message [10531.900683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10534.101637] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10543.380121] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10552.293777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10552.312787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.318491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.333131] Lustre: Skipped 3 previous similar messages [10552.351800] Lustre: Skipped 1 previous similar message [10555.692230] Lustre: server umount lustre-OST0000 complete [10562.019338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10562.027142] LustreError: Skipped 1 previous similar message [10562.030208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.047946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10566.575090] Lustre: server umount lustre-MDT0000 complete [10567.655602] LustreError: 366029:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.679341] LustreError: 366029:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10571.027107] LustreError: 366008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878877 with bad export cookie 11179407203359904842 [10571.033645] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10571.038636] LustreError: 366008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10571.467618] Lustre: server umount lustre-MDT0001 complete [10585.034824] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10587.909190] Key type lgssc unregistered [10588.212835] LNet: 369579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10588.219593] LNetError: 369579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10588.233681] LNet: Removed LNI 192.168.206.136@tcp [10589.185707] Key type .llcrypt unregistered [10589.189391] Key type ._llcrypt unregistered [10617.240612] Key type ._llcrypt registered [10617.243870] Key type .llcrypt registered [10617.353691] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10618.308731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10618.347125] alg: No test for adler32 (adler32-zlib) [10619.413413] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [10619.692376] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10621.344136] Key type lgssc registered [10622.349683] Lustre: Echo OBD driver; http://www.lustre.org/ [10629.767610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10629.784175] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.791458] LustreError: 371026:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10629.806259] LustreError: 371026:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10629.817520] LustreError: 371026:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10629.824767] LustreError: 371026:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10629.836566] LustreError: 371026:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10636.636400] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10636.644075] LustreError: 371423:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10636.672568] LustreError: 371423:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10636.685153] LustreError: 371423:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10636.693637] LustreError: 371423:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10636.700651] LustreError: 371423:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10655.345512] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10657.884752] Key type lgssc unregistered [10658.197756] LNet: 372430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10658.212219] LNetError: 372430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10658.240189] LNet: Removed LNI 192.168.206.136@tcp [10659.102235] Key type .llcrypt unregistered [10659.104477] Key type ._llcrypt unregistered [10676.376920] Key type ._llcrypt registered [10676.380390] Key type .llcrypt registered [10676.498598] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [10691.818979] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10693.112842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10693.515957] alg: No test for adler32 (adler32-zlib) [10694.769586] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [10695.105869] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10696.848257] Key type lgssc registered [10698.356249] Lustre: Echo OBD driver; http://www.lustre.org/ [10733.863807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10733.913042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.183609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10735.258745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10735.321476] Lustre: lustre-MDT0000: new disk, initializing [10735.380586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10735.398556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10740.053579] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10751.579599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.698785] Lustre: 375587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.773458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10751.779956] Lustre: Skipped 1 previous similar message [10751.794640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10751.870821] Lustre: lustre-MDT0001: new disk, initializing [10751.941785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10751.977095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10751.989815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10756.886855] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10766.438962] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10771.769804] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10778.720695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.913246] Lustre: lustre-OST0000: new disk, initializing [10778.916564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10778.922415] Lustre: 377553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10779.000092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10784.767587] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10788.433251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10788.447223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10788.549927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10794.142871] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10798.486787] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10798.725299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10803.032697] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10803.199596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10809.328684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.332298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10809.350726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10809.357982] LustreError: Skipped 1 previous similar message [10811.236173] Lustre: server umount lustre-OST0000 complete [10818.530378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10818.543840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.561188] Lustre: Skipped 1 previous similar message [10818.568524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.573387] Lustre: Skipped 1 previous similar message [10819.558065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.674284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10824.684654] Lustre: Skipped 3 previous similar messages [10829.280335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10829.775081] Lustre: server umount lustre-MDT0000 complete [10833.657369] LustreError: 375579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879140 with bad export cookie 3825559821626903519 [10833.667532] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.676715] LustreError: 375579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10834.225265] Lustre: server umount lustre-MDT0001 complete [10843.726707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10845.857683] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:05:50 (1787879150) [10852.827484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.485191] LustreError: 380012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.509214] LustreError: 380012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10853.622204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10858.251829] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10858.982324] LustreError: 380013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.102683] LustreError: 380012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.679727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.996599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10871.493361] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10880.918696] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10888.111853] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10895.435061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10895.917678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10902.086430] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10910.287547] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10917.191658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.348659] Lustre: lustre-OST0001: new disk, initializing [10917.352858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10917.358094] Lustre: 382850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10917.450878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10923.327282] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10926.709777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10926.718453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10926.813499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10933.400755] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10940.805600] Lustre: lustre-OST0001: Not available for connect from 192.168.206.36@tcp (stopping) [10942.438114] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.441522] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10942.455718] Lustre: Skipped 1 previous similar message [10942.474746] LustreError: Skipped 1 previous similar message [10944.252881] Lustre: server umount lustre-OST0001 complete [10945.866828] LustreError: 381835:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.682206] LustreError: 381837:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.707494] LustreError: 381837:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10962.401489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10962.413370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10962.456412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10962.470572] Lustre: Skipped 2 previous similar messages [10962.920544] LustreError: 383774:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.963726] LustreError: 383774:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10976.224113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10976.475496] Lustre: server umount lustre-OST0000 complete [10983.396904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.404196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.425987] Lustre: Skipped 2 previous similar messages [10983.452259] Lustre: Skipped 6 previous similar messages [10987.555768] Lustre: server umount lustre-MDT0000 complete [10988.514212] LustreError: 380740:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.541516] LustreError: 380740:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10992.781190] LustreError: 379992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879299 with bad export cookie 3825559821626904408 [10992.783214] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10992.796718] LustreError: 379992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10999.381398] Lustre: server umount lustre-MDT0001 complete [11011.465408] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11014.254371] Key type lgssc unregistered [11014.446554] LNet: 384842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11014.452896] LNetError: 384842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11014.468701] LNet: Removed LNI 192.168.206.136@tcp [11015.500929] Key type .llcrypt unregistered [11015.505834] Key type ._llcrypt unregistered [11066.211323] Key type ._llcrypt registered [11066.213116] Key type .llcrypt registered [11066.295264] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11067.390700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11067.419036] alg: No test for adler32 (adler32-zlib) [11068.440461] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11068.683929] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [11070.329657] Key type lgssc registered [11071.336665] Lustre: Echo OBD driver; http://www.lustre.org/ [11078.605129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11078.633353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.933926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11079.969697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11080.240348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11085.936827] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11094.583101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.720350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11094.756213] Lustre: 387639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11095.041940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11099.560890] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11108.745513] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.208728] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11117.542594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.564930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.580479] Lustre: Skipped 1 previous similar message [11122.658034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.666508] Lustre: Skipped 1 previous similar message [11123.849069] Lustre: server umount lustre-MDT0000 complete [11127.642305] LustreError: 387632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879434 with bad export cookie 16562050583993113714 [11127.648899] LustreError: 387632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11127.654174] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11127.883404] Lustre: server umount lustre-MDT0001 complete [11138.251478] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:10:42 (1787879442) [11143.623028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.033259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11148.598658] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11149.288660] LustreError: 390154:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.320063] LustreError: 390154:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11154.402444] LustreError: 390155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.129230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11157.482192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.590464] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11170.237829] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11176.052643] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11183.134420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11183.278718] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11183.401466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11188.638876] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11196.753252] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11206.313631] Lustre: Permanently deactivating lustre-OST0000 [11206.326251] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11216.446967] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11217.746284] LustreError: 393192:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11217.757638] LustreError: 393192:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11217.768901] LustreError: 393192:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11221.291100] LustreError: 393242:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11221.303428] LustreError: 393242:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11229.624173] Lustre: server umount lustre-OST0000 complete [11232.453851] LustreError: 393491:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11232.458866] LustreError: 393491:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11239.396474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.397114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.426230] Lustre: Skipped 1 previous similar message [11239.440393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.449103] Lustre: Skipped 1 previous similar message [11244.514092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11244.521413] Lustre: Skipped 1 previous similar message [11245.627401] Lustre: server umount lustre-MDT0000 complete [11249.317629] LustreError: 392848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879556 with bad export cookie 16562050583993114288 [11249.323201] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11249.325834] LustreError: 392848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11260.088448] Lustre: server umount lustre-MDT0001 complete [11268.217148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.950868] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11275.661940] LustreError: 394651:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11281.567300] Lustre: 394990:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.136@tcp [11283.345411] Lustre: server umount MGS complete [11295.735833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.404884] LustreError: 395715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.447162] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11296.458027] Lustre: Skipped 1 previous similar message [11296.595936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11301.858426] LustreError: 395716:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.078443] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11306.980610] LustreError: 395715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.365021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.566181] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11314.547110] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11322.271734] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11328.739540] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11330.738626] Lustre: Permanently reactivating lustre-OST0000 [11330.749128] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11330.753320] Lustre: Skipped 2 previous similar messages [11360.738072] LustreError: 397545:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11370.119734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11370.447634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.455288] Lustre: Skipped 1 previous similar message [11377.772126] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11389.524913] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11401.185737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11401.189361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.200035] LustreError: Skipped 1 previous similar message [11401.205317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11401.218837] Lustre: Skipped 2 previous similar messages [11401.250894] Lustre: Skipped 1 previous similar message [11410.400986] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11410.594167] Lustre: server umount lustre-OST0000 complete [11416.552108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.558242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11416.565431] Lustre: Skipped 1 previous similar message [11416.576588] Lustre: Skipped 3 previous similar messages [11421.592447] Lustre: server umount lustre-MDT0000 complete [11421.682424] LustreError: 396443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.707909] LustreError: 396443:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11426.680563] LustreError: 399473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879733 with bad export cookie 16562050583993115345 [11426.683948] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11426.696097] LustreError: 399473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11427.093870] Lustre: server umount lustre-MDT0001 complete [11441.009751] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11443.649877] Key type lgssc unregistered [11443.961668] LNet: 400229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11443.973556] LNetError: 400229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11443.988704] LNet: Removed LNI 192.168.206.136@tcp [11445.043162] Key type .llcrypt unregistered [11445.046433] Key type ._llcrypt unregistered [11464.189428] Key type ._llcrypt registered [11464.192417] Key type .llcrypt registered [11464.283447] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [11480.776473] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11481.654232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11481.687224] alg: No test for adler32 (adler32-zlib) [11482.874878] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11483.112684] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [11484.840226] Key type lgssc registered [11486.096377] Lustre: Echo OBD driver; http://www.lustre.org/ [11517.624476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11517.676863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.896705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11518.932629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11519.001277] Lustre: lustre-MDT0000: new disk, initializing [11519.065328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11519.077565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11523.492664] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11533.868637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.011152] Lustre: 403386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11534.100240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11534.108618] Lustre: Skipped 1 previous similar message [11534.214476] Lustre: lustre-MDT0001: new disk, initializing [11534.345406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11534.421273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11534.433742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11538.959674] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11548.596828] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11554.187725] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11560.386453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11560.573792] Lustre: lustre-OST0000: new disk, initializing [11560.577415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11560.583700] Lustre: 405354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11560.682650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11565.409956] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11567.194479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11567.217923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11567.316261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11574.534078] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11578.892157] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11579.052723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11583.099906] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11583.285503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11588.064960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11588.074986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.078031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11588.079578] LustreError: Skipped 1 previous similar message [11588.106436] Lustre: Skipped 1 previous similar message [11593.194192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11593.201332] Lustre: Skipped 2 previous similar messages [11598.307807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11598.313555] Lustre: Skipped 1 previous similar message [11599.328230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11599.575834] Lustre: server umount lustre-OST0000 complete [11603.426651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11603.427424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11603.442835] Lustre: Skipped 1 previous similar message [11608.546853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11608.557135] Lustre: Skipped 2 previous similar messages [11609.791661] Lustre: server umount lustre-MDT0000 complete [11613.359637] LustreError: 403379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879920 with bad export cookie 10677476319317504397 [11613.361206] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11613.369181] LustreError: 403379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11613.880908] Lustre: server umount lustre-MDT0001 complete [11624.979508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11626.752173] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:18:51 (1787879931) [11633.553978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.134561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11638.970493] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11639.271675] LustreError: 407817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.289838] LustreError: 407817:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11643.365744] LustreError: 407816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.444765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.485248] LustreError: 407817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.751440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11653.081505] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11659.802558] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11665.690849] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11673.738336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.149849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11681.289798] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11690.631844] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11692.510950] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11695.083060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11703.782793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11703.789305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11703.791932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11703.814194] Lustre: Skipped 1 previous similar message [11706.310583] Lustre: server umount lustre-OST0000 complete [11710.433673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11710.452943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11724.256790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11724.269376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11724.276293] Lustre: Skipped 7 previous similar messages [11724.467806] Lustre: server umount lustre-MDT0000 complete [11728.123638] LustreError: 408550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880034 with bad export cookie 10677476319317505272 [11728.128578] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.145640] LustreError: 408550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11728.442903] Lustre: server umount lustre-MDT0001 complete [11739.866954] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11742.807586] Key type lgssc unregistered [11743.095903] LNet: 411546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.105477] LNetError: 411546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11743.128937] LNet: Removed LNI 192.168.206.136@tcp [11744.007576] Key type .llcrypt unregistered [11744.010604] Key type ._llcrypt unregistered [11767.655189] Key type ._llcrypt registered [11767.658593] Key type .llcrypt registered [11767.757499] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11768.078246] Key type .llcrypt unregistered [11768.082909] Key type ._llcrypt unregistered [11787.434560] Key type ._llcrypt registered [11787.438725] Key type .llcrypt registered [11787.551918] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11788.970422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11789.020131] alg: No test for adler32 (adler32-zlib) [11790.028321] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11790.276245] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [11791.960870] Key type lgssc registered [11793.055335] Lustre: Echo OBD driver; http://www.lustre.org/ [11801.152456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11801.167758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.723054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11807.564340] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11812.840765] LustreError: 413763:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.856779] LustreError: 413763:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11814.950605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11815.190752] LustreError: 413764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.202406] LustreError: 413764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11815.208990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11819.929361] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11820.515746] LustreError: 413759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.531676] LustreError: 413759:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11825.638082] LustreError: 413759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11825.651707] LustreError: 413759:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11825.666569] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11827.705092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11832.201151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.535178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11836.773909] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11847.211991] Lustre: server umount lustre-OST0000 complete [11852.777046] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11852.782533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.795806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11858.401471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11858.408817] Lustre: Skipped 3 previous similar messages [11861.533644] Lustre: server umount lustre-MDT0000 complete [11863.456131] Lustre: 413074:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787880154/real 1787880154] req@00000000cced1393 x1874728160617344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787880170 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11863.480958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.491808] Lustre: Skipped 1 previous similar message [11863.495586] LustreError: 413764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.511762] LustreError: 413764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11864.481330] LustreError: 413745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880171 with bad export cookie 9863285014180550858 [11864.485300] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11864.494030] LustreError: 413745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11864.754343] Lustre: server umount lustre-MDT0001 complete [11872.881730] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11875.376465] Key type lgssc unregistered [11875.614267] LNet: 416917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11875.623660] LNetError: 416917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11875.633841] LNet: Removed LNI 192.168.206.136@tcp [11876.404192] Key type .llcrypt unregistered [11876.406402] Key type ._llcrypt unregistered [11901.043751] Key type ._llcrypt registered [11901.046732] Key type .llcrypt registered [11901.146026] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11902.444889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11902.474164] alg: No test for adler32 (adler32-zlib) [11903.548595] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11903.888094] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [11905.592145] Key type lgssc registered [11906.795255] Lustre: Echo OBD driver; http://www.lustre.org/ [11915.908650] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11915.942276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11931.488396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11936.608565] LustreError: 418310:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [11963.232200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11968.492745] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11974.428050] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11980.429724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.989497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11981.034608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11985.038958] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11986.403611] LustreError: 418336:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.425094] LustreError: 418336:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11991.522195] LustreError: 418335:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.140898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11993.598151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11997.504833] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12004.009578] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12009.465028] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12017.121394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12017.135772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.148186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12018.149015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12018.154338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.681315] Lustre: server umount lustre-OST0000 complete [12023.273806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.279369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12023.289926] Lustre: Skipped 1 previous similar message [12023.308775] Lustre: Skipped 2 previous similar messages [12024.277489] Lustre: server umount lustre-MDT0000 complete [12027.270275] LustreError: 418316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880334 with bad export cookie 877810715764748782 [12027.276387] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12027.277682] LustreError: 418316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12027.499923] Lustre: server umount lustre-MDT0001 complete [12037.605763] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [12040.254456] Key type lgssc unregistered [12040.675651] LNet: 421881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12040.688258] LNetError: 421881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12040.716678] LNet: Removed LNI 192.168.206.136@tcp [12041.796091] Key type .llcrypt unregistered [12041.797908] Key type ._llcrypt unregistered [12070.186545] Key type ._llcrypt registered [12070.188060] Key type .llcrypt registered [12070.271824] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [12071.111403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12071.211473] alg: No test for adler32 (adler32-zlib) [12072.342918] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12072.616222] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [12074.296269] Key type lgssc registered [12075.449900] Lustre: Echo OBD driver; http://www.lustre.org/ [12082.184586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12082.221163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.594907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12087.595975] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12093.935529] LustreError: 423298:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.958325] LustreError: 423298:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12095.656789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.049831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12100.005431] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12106.853511] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12112.521544] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12118.863513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12119.187833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12124.753550] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12128.441853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12131.547088] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12136.929651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12136.934036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12136.940597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12138.337909] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [12139.005440] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.018069] Lustre: Skipped 1 previous similar message [12141.470567] Lustre: server umount lustre-MDT0000 complete [12143.452342] LustreError: 423292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12144.950088] LustreError: 423279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.36@tcp arrived at 1787880451 with bad export cookie 17765437064368650889 [12147.439885] LustreError: 423277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880454 with bad export cookie 17765437064368649993 [12147.460503] LustreError: 423277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12147.465060] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12149.222160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.225237] LustreError: 424026:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.243523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.251115] LustreError: 424026:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12149.269680] Lustre: Skipped 3 previous similar messages [12152.293977] LustreError: 423298:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.294683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12152.307155] LustreError: 423298:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12153.748859] Lustre: server umount lustre-OST0000 complete [12157.412268] LustreError: 424026:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.475914] LustreError: 424026:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12163.591590] Lustre: server umount lustre-MDT0001 complete [12177.680061] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [12180.041492] Key type lgssc unregistered [12180.245744] LNet: 426984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12180.255646] LNetError: 426984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12180.290888] LNet: Removed LNI 192.168.206.136@tcp [12181.090741] Key type .llcrypt unregistered [12181.096794] Key type ._llcrypt unregistered [12205.492669] Key type ._llcrypt registered [12205.500761] Key type .llcrypt registered [12205.618449] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [12207.050665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12207.087090] alg: No test for adler32 (adler32-zlib) [12208.053822] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12208.337595] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [12210.080769] Key type lgssc registered [12211.478762] Lustre: Echo OBD driver; http://www.lustre.org/ [12219.776743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12219.788139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12221.136849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12225.366533] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12230.643899] LustreError: 428401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.678921] LustreError: 428401:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12234.591730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12234.997935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12239.579920] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12247.310451] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12253.806356] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12263.698604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12264.382725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12265.398670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12271.477200] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12282.285746] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12290.027404] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12290.059128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12290.065232] Lustre: Skipped 1 previous similar message [12293.841861] Lustre: server umount lustre-MDT0001 complete [12295.139689] LustreError: 429934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12295.165627] LustreError: 429934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12300.266238] LustreError: 429129:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12300.291769] LustreError: 429129:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12301.285934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12301.300689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.309368] Lustre: Skipped 1 previous similar message [12301.313881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.316059] Lustre: Skipped 1 previous similar message [12305.382922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12305.383430] LustreError: 428401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.057355] Lustre: server umount lustre-OST0000 complete [12310.498888] LustreError: 428402:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.311462] Lustre: server umount lustre-MDT0000 complete [12328.416805] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [12330.992877] Key type lgssc unregistered [12331.240140] LNet: 432095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12331.257751] LNetError: 432095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12331.278920] LNet: Removed LNI 192.168.206.136@tcp [12332.107970] Key type .llcrypt unregistered [12332.110728] Key type ._llcrypt unregistered [12359.613475] Key type ._llcrypt registered [12359.615594] Key type .llcrypt registered [12359.687862] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [12375.562861] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [12376.422802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12376.442577] alg: No test for adler32 (adler32-zlib) [12377.438544] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12377.626505] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [12379.288182] Key type lgssc registered [12380.331455] Lustre: Echo OBD driver; http://www.lustre.org/ [12415.431847] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [12429.380532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12429.396088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.590309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12430.623791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12430.758879] Lustre: lustre-MDT0000: new disk, initializing [12430.865438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12430.886733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12434.647648] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12455.080288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12455.301202] Lustre: lustre-OST0000: new disk, initializing [12455.303748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12455.306717] Lustre: Skipped 1 previous similar message [12455.313180] Lustre: 437126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12455.374238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.262051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12465.275486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12465.310967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12465.652572] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12475.278141] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12492.284130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.400693] Lustre: 436108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12492.484141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12492.574830] Lustre: lustre-MDT0001: new disk, initializing [12492.732616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12492.836096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12492.848146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12497.081452] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12505.390661] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12505.539304] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12509.465033] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12509.623357] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12515.813708] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.817252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12515.826801] Lustre: Skipped 1 previous similar message [12515.842225] Lustre: Skipped 1 previous similar message [12518.751843] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [12520.269613] Lustre: server umount lustre-MDT0001 complete [12526.053262] LustreError: 436115:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.069526] LustreError: 436115:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12531.176807] LustreError: 436121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.187101] LustreError: 436121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12536.299341] LustreError: 436116:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.316672] LustreError: 436116:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12541.411526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12541.413703] LustreError: 436121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12541.426433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12543.316956] Lustre: server umount lustre-OST0000 complete [12553.020097] Lustre: server umount lustre-MDT0000 complete [12568.761768] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [12571.512742] Key type lgssc unregistered [12571.771402] LNet: 440877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12571.791363] LNetError: 440877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12571.814979] LNet: Removed LNI 192.168.206.136@tcp [12572.774151] Key type .llcrypt unregistered [12572.775829] Key type ._llcrypt unregistered [12613.508941] Key type ._llcrypt registered [12613.511289] Key type .llcrypt registered [12613.599312] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [12613.918417] Key type .llcrypt unregistered [12613.930845] Key type ._llcrypt unregistered [12631.651514] Key type ._llcrypt registered [12631.653711] Key type .llcrypt registered [12631.791089] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [12633.852584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12633.936622] alg: No test for adler32 (adler32-zlib) [12635.229606] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12635.540925] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [12637.304170] Key type lgssc registered [12639.109188] Lustre: Echo OBD driver; http://www.lustre.org/ [12666.434616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12666.449433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.652887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12667.681636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12667.759451] Lustre: lustre-MDT0000: new disk, initializing [12667.858099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12667.881322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12673.292967] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12684.321044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.519746] Lustre: 444260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12684.606224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12684.616355] Lustre: Skipped 1 previous similar message [12684.766789] Lustre: lustre-MDT0001: new disk, initializing [12684.900894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12684.933498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12684.948411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12689.184401] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12700.431664] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12706.836739] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12713.911343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12714.344322] Lustre: lustre-OST0000: new disk, initializing [12714.361500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12714.379768] Lustre: 446227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12714.526762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.691138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12719.710269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12719.840557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12721.102330] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12730.406896] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12735.969533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12735.974847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.986715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12740.578939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12740.586652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12740.600860] Lustre: Skipped 1 previous similar message [12740.613191] Lustre: Skipped 2 previous similar messages [12742.186616] Lustre: server umount lustre-MDT0000 complete [12745.696880] LustreError: 444252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881052 with bad export cookie 15793193828292347121 [12745.729949] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12745.795526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.824589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12746.145772] Lustre: server umount lustre-MDT0001 complete [12756.016367] Lustre: server umount lustre-OST0000 complete [12769.866189] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:37:54 (1787881074) [12778.131960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.720081] LustreError: 448256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.760793] LustreError: 448256:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12778.831887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12783.370354] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12784.100822] LustreError: 448257:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.223891] LustreError: 448256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.410798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12792.825443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12797.707640] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12804.890235] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12809.922971] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12829.627058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12829.904894] Lustre: Found index 0 for lustre-OST0000, updating log [12830.072855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12836.395933] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [12844.199815] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12849.531063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12859.881425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12859.882719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12859.901396] LustreError: Skipped 1 previous similar message [12859.904600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12859.918380] Lustre: Skipped 1 previous similar message [12859.941245] Lustre: Skipped 1 previous similar message [12867.057291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12867.066840] Lustre: Skipped 3 previous similar messages [12870.112127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12870.271580] Lustre: server umount lustre-OST0000 complete [12886.099052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12886.102423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12886.123544] Lustre: Skipped 1 previous similar message [12886.140805] Lustre: Skipped 1 previous similar message [12892.558091] Lustre: server umount lustre-MDT0000 complete [12896.133242] LustreError: 448238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881202 with bad export cookie 15793193828292348514 [12896.148237] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12896.153775] LustreError: 448238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12896.638390] Lustre: server umount lustre-MDT0001 complete [12899.744723] LNet: 1 local NIs in recovery (showing 1): 192.168.206.136@tcp [12905.594438] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:40:10 (1787881210) [13528.902634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13529.244843] LustreError: 454670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13529.312686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13533.791505] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13534.692094] LustreError: 454671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.812087] LustreError: 454670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.977211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13542.298619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13547.048322] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13554.730896] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13560.429827] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13580.977403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13581.317455] Lustre: Found index 0 for lustre-OST0000, updating log [13581.575814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13588.400456] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13589.985218] LNet: 1 local NIs in recovery (showing 1): 192.168.206.136@tcp [13596.873167] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13599.461230] Lustre: server umount lustre-OST0000 complete [13603.809783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13603.823705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13603.851049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13603.860493] Lustre: Skipped 2 previous similar messages [13604.898941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13605.991698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13606.007208] Lustre: Skipped 3 previous similar messages [13609.574366] Lustre: server umount lustre-MDT0000 complete [13610.145623] LustreError: 455398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.161257] LustreError: 455398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13613.242349] LustreError: 454650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881919 with bad export cookie 15793193828292349599 [13613.246797] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13613.257468] LustreError: 454650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13613.762636] Lustre: server umount lustre-MDT0001 complete [13645.344630] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [13655.821761] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [13693.222643] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:53:17 (1787881997) [13710.100900] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [13712.779323] Key type lgssc unregistered [13713.057719] LNet: 461288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13713.066780] LNetError: 461288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13842.086447] LNet: Removed LNI 192.168.206.136@tcp [13843.022514] Key type .llcrypt unregistered [13843.025811] Key type ._llcrypt unregistered [13863.209736] Key type ._llcrypt registered [13863.211497] Key type .llcrypt registered [13863.383812] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [13864.345455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13864.368039] alg: No test for adler32 (adler32-zlib) [13865.484606] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [13865.675900] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [13867.344199] Key type lgssc registered [13868.719701] Lustre: Echo OBD driver; http://www.lustre.org/ [13873.173145] LNet: Added LNI 192.168.206.136@tcp100 [8/256/0/180] [13874.072607] LNet: Added LNI 192.168.206.136@tcp101 [8/256/0/180] [13874.956390] LNet: Added LNI 192.168.206.136@tcp102 [8/256/0/180] [13875.934434] LNet: Added LNI 192.168.206.136@tcp103 [8/256/0/180] [13876.840976] LNet: Added LNI 192.168.206.136@tcp104 [8/256/0/180] [13877.670554] LNet: Added LNI 192.168.206.136@tcp105 [8/256/0/180] [13878.545082] LNet: Added LNI 192.168.206.136@tcp106 [8/256/0/180] [13879.447035] LNet: Added LNI 192.168.206.136@tcp107 [8/256/0/180] [13880.215912] LNet: Added LNI 192.168.206.136@tcp108 [8/256/0/180] [13880.879595] LNet: Added LNI 192.168.206.136@tcp109 [8/256/0/180] [13881.711342] LNet: Added LNI 192.168.206.136@tcp110 [8/256/0/180] [13882.593441] LNet: Added LNI 192.168.206.136@tcp111 [8/256/0/180] [13883.321605] LNet: Added LNI 192.168.206.136@tcp112 [8/256/0/180] [13884.172707] LNet: Added LNI 192.168.206.136@tcp113 [8/256/0/180] [13885.045238] LNet: Added LNI 192.168.206.136@tcp114 [8/256/0/180] [13885.900717] LNet: Added LNI 192.168.206.136@tcp115 [8/256/0/180] [13886.898931] LNet: Added LNI 192.168.206.136@tcp116 [8/256/0/180] [13887.800670] LNet: Added LNI 192.168.206.136@tcp117 [8/256/0/180] [13888.660621] LNet: Added LNI 192.168.206.136@tcp118 [8/256/0/180] [13889.552590] LNet: Added LNI 192.168.206.136@tcp119 [8/256/0/180] [13890.599708] LNet: Added LNI 192.168.206.136@tcp120 [8/256/0/180] [13891.618056] LNet: Added LNI 192.168.206.136@tcp121 [8/256/0/180] [13892.462265] LNet: Added LNI 192.168.206.136@tcp122 [8/256/0/180] [13893.381563] LNet: Added LNI 192.168.206.136@tcp123 [8/256/0/180] [13894.383495] LNet: Added LNI 192.168.206.136@tcp124 [8/256/0/180] [13895.276089] LNet: Added LNI 192.168.206.136@tcp125 [8/256/0/180] [13896.173410] LNet: Added LNI 192.168.206.136@tcp126 [8/256/0/180] [13897.182571] LNet: Added LNI 192.168.206.136@tcp127 [8/256/0/180] [13898.280549] LNet: Added LNI 192.168.206.136@tcp128 [8/256/0/180] [13899.305183] LNet: Added LNI 192.168.206.136@tcp129 [8/256/0/180] [13900.305359] LNet: Added LNI 192.168.206.136@tcp130 [8/256/0/180] [13901.244539] LNet: Added LNI 192.168.206.136@tcp131 [8/256/0/180] [13902.385964] LNet: Added LNI 192.168.206.136@tcp132 [8/256/0/180] [13903.238441] LNet: Added LNI 192.168.206.136@tcp133 [8/256/0/180] [13904.133604] LNet: Added LNI 192.168.206.136@tcp134 [8/256/0/180] [13905.052645] LNet: Added LNI 192.168.206.136@tcp135 [8/256/0/180] [13912.618968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13912.657974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13913.830101] Lustre: 464325:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.136@tcp: can't add NID 192.168.206.136@tcp130: rc = -75 [13913.921072] LNetError: 464352:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13913.937184] LNetError: Protocol error connecting to 192.168.206.136@tcp119 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13913.967071] LNetError: Skipped 3 previous similar messages [13914.086604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13914.139862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13914.221746] Lustre: lustre-MDT0000: new disk, initializing [13914.276110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13914.289668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13914.848261] LNet: 8 local NIs in recovery (showing 8): 192.168.206.136@tcp122, 192.168.206.136@tcp123, 192.168.206.136@tcp119, 192.168.206.136@tcp124, 192.168.206.136@tcp129, 192.168.206.136@tcp130, 192.168.206.136@tcp131, 192.168.206.136@tcp132 [13914.851900] LNetError: 464350:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13914.903622] LNetError: 464350:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [13914.917769] LNetError: Protocol error connecting to 192.168.206.136@tcp122 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13914.934767] LNetError: Skipped 10 previous similar messages [13915.937547] LNetError: 464356:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13915.961709] LNetError: Protocol error connecting to 192.168.206.136@tcp130 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13917.996694] LNetError: 461784:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13918.005635] LNetError: 461784:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13918.013200] LNetError: Protocol error connecting to 192.168.206.136@tcp130 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13918.023812] LNetError: Skipped 1 previous similar message [13918.995984] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13928.236283] LNetError: 464347:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13928.255353] LNetError: 464347:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13928.269874] LNetError: Protocol error connecting to 192.168.206.136@tcp122 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13928.300233] LNetError: Skipped 1 previous similar message [13929.236901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13929.398729] Lustre: 464341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13929.473718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13929.478945] Lustre: Skipped 1 previous similar message [13929.534287] Lustre: lustre-MDT0001: new disk, initializing [13929.570355] Lustre: 465144:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.136@tcp: can't add NID 192.168.206.136@tcp130: rc = -75 [13929.577256] Lustre: 465144:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13929.721477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13929.754227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13929.776752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13931.316962] Lustre: 465226:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.136@tcp: can't add NID 192.168.206.136@tcp130: rc = -75 [13931.325521] Lustre: 465226:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13934.437037] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13936.440967] LNetError: 461785:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13936.448272] LNetError: 461785:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [13936.453714] LNetError: Protocol error connecting to 192.168.206.136@tcp133 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13936.460634] LNetError: Skipped 11 previous similar messages [13945.083275] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13951.738560] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13958.769188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13958.881683] LNetError: 464346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13958.894797] LNetError: 464346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [13958.906168] LNetError: Protocol error connecting to 192.168.206.136@tcp120 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13958.920665] LNetError: Skipped 11 previous similar messages [13959.008140] Lustre: lustre-OST0000: new disk, initializing [13959.011589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13959.019834] Lustre: 466319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13959.093152] Lustre: 466324:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.136@tcp: can't add NID 192.168.206.136@tcp130: rc = -75 [13959.103182] Lustre: 466324:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13959.159156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13965.406686] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [13967.458180] Lustre: 466776:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.136@tcp: can't add NID 192.168.206.136@tcp130: rc = -75 [13967.465669] Lustre: 466776:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13967.548438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13967.558576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13967.703589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13974.304437] LNet: 4 local NIs in recovery (showing 4): 192.168.206.136@tcp133, 192.168.206.136@tcp134, 192.168.206.136@tcp130, 192.168.206.136@tcp122 [13975.343792] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13988.322578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13988.325710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13988.344846] LustreError: Skipped 1 previous similar message [13988.349789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13988.396705] Lustre: Skipped 1 previous similar message [13990.033358] Lustre: server umount lustre-OST0000 complete [13992.737811] LNetError: 461784:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.136 [13992.762720] LNetError: 461784:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [13992.777779] LNetError: Protocol error connecting to 192.168.206.136@tcp133 at host 192.168.206.136:988: is it running a compatible version of Lustre? [13992.787719] LNetError: Skipped 21 previous similar messages [13996.518486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13996.524493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13996.532649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13996.536627] Lustre: Skipped 1 previous similar message [13998.561694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13998.571435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14000.732115] Lustre: server umount lustre-MDT0000 complete [14004.753328] LustreError: 465143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882311 with bad export cookie 921866352464669565 [14004.757131] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14004.762929] LustreError: 465143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14005.080650] Lustre: server umount lustre-MDT0001 complete [14017.362472] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [14020.133790] Key type lgssc unregistered [14020.455424] LNet: 468235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14020.461507] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14020.469211] LNet: Removed LNI 192.168.206.136@tcp [14021.536143] LNet: Removed LNI 192.168.206.136@tcp100 [14021.551761] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14021.559793] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14021.567042] LNet: Removed LNI 192.168.206.136@tcp101 [14022.624144] LNet: Removed LNI 192.168.206.136@tcp102 [14022.650502] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14022.665265] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14022.675402] LNet: Removed LNI 192.168.206.136@tcp103 [14023.712153] LNet: Removed LNI 192.168.206.136@tcp104 [14024.736185] LNet: Removed LNI 192.168.206.136@tcp105 [14024.743531] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14024.754117] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14024.769248] LNet: Removed LNI 192.168.206.136@tcp106 [14025.824200] LNet: Removed LNI 192.168.206.136@tcp107 [14026.848096] LNet: Removed LNI 192.168.206.136@tcp108 [14027.874773] LNet: Removed LNI 192.168.206.136@tcp109 [14028.896171] LNet: Removed LNI 192.168.206.136@tcp110 [14028.903199] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14028.911192] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14029.920899] LNet: Removed LNI 192.168.206.136@tcp111 [14030.947021] LNet: Removed LNI 192.168.206.136@tcp112 [14031.968701] LNet: Removed LNI 192.168.206.136@tcp113 [14032.992116] LNet: Removed LNI 192.168.206.136@tcp114 [14034.016148] LNet: Removed LNI 192.168.206.136@tcp115 [14035.040261] LNet: Removed LNI 192.168.206.136@tcp116 [14036.064140] LNet: Removed LNI 192.168.206.136@tcp117 [14037.088244] LNet: Removed LNI 192.168.206.136@tcp118 [14037.112666] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14037.121180] LNetError: 468235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14037.132916] LNet: Removed LNI 192.168.206.136@tcp119 [14038.176384] LNet: Removed LNI 192.168.206.136@tcp120 [14039.200138] LNet: Removed LNI 192.168.206.136@tcp121 [14039.204597] LNet: Removed LNI 192.168.206.136@tcp122 [14040.224125] LNet: Removed LNI 192.168.206.136@tcp123 [14041.248163] LNet: Removed LNI 192.168.206.136@tcp124 [14042.272988] LNet: Removed LNI 192.168.206.136@tcp125 [14043.300304] LNet: Removed LNI 192.168.206.136@tcp126 [14044.320145] LNet: Removed LNI 192.168.206.136@tcp127 [14045.344147] LNet: Removed LNI 192.168.206.136@tcp128 [14046.368280] LNet: Removed LNI 192.168.206.136@tcp129 [14046.383276] LNet: Removed LNI 192.168.206.136@tcp130 [14047.394582] LNet: Removed LNI 192.168.206.136@tcp131 [14048.416169] LNet: Removed LNI 192.168.206.136@tcp132 [14048.418421] LNet: Removed LNI 192.168.206.136@tcp133 [14049.441557] LNet: Removed LNI 192.168.206.136@tcp134 [14050.474183] LNet: Removed LNI 192.168.206.136@tcp135 [14051.437077] Key type .llcrypt unregistered [14051.441369] Key type ._llcrypt unregistered [14079.111730] Key type ._llcrypt registered [14079.113492] Key type .llcrypt registered [14079.231967] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [14095.924785] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [14097.162319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14097.211706] alg: No test for adler32 (adler32-zlib) [14098.220908] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [14098.513279] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [14100.282281] Key type lgssc registered [14101.542112] Lustre: Echo OBD driver; http://www.lustre.org/ [14141.272944] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:00:45 (1787882445) [14161.872696] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [14164.236803] Key type lgssc unregistered [14164.617603] LNet: 472339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14164.623373] LNetError: 472339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14164.643819] LNet: Removed LNI 192.168.206.136@tcp [14165.448907] Key type .llcrypt unregistered [14165.451851] Key type ._llcrypt unregistered [14184.903867] Key type ._llcrypt registered [14184.905943] Key type .llcrypt registered [14185.022540] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [14186.355459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14186.407112] alg: No test for adler32 (adler32-zlib) [14187.443616] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [14187.869233] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [14189.664369] Key type lgssc registered [14190.952284] Lustre: Echo OBD driver; http://www.lustre.org/ [14201.028135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14201.058879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14202.435513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14202.506206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14202.752700] Lustre: lustre-MDT0000: new disk, initializing [14202.949594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14202.989144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14207.924837] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14219.962622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14220.111561] Lustre: 473554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14220.185696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14220.188610] Lustre: Skipped 1 previous similar message [14220.298516] Lustre: lustre-MDT0001: new disk, initializing [14220.410516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14220.456118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14220.469644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14225.869453] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14236.274447] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14242.054712] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14249.355771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14249.630150] Lustre: lustre-OST0000: new disk, initializing [14249.637828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14249.647230] Lustre: 475522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14249.737816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14256.433698] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14258.247271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14258.270098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14258.412568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14267.293249] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14271.725563] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing lnet_if_list [14277.647748] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14277.672312] LNet: Added LNI 192.168.206.136@tcp10 [8/256/0/180] [14283.858544] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14283.898216] LNet: Added LNI 192.168.206.136@tcp11 [8/256/0/180] [14288.256373] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14288.278231] LNet: Added LNI 192.168.206.136@tcp12 [8/256/0/180] [14292.936931] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14292.970800] LNet: Added LNI 192.168.206.136@tcp13 [8/256/0/180] [14297.953323] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14297.975726] LNet: Added LNI 192.168.206.136@tcp14 [8/256/0/180] [14303.085955] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14303.114513] LNet: Added LNI 192.168.206.136@tcp15 [8/256/0/180] [14308.039846] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14308.067643] LNet: Added LNI 192.168.206.136@tcp16 [8/256/0/180] [14313.299299] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14313.334669] LNet: Added LNI 192.168.206.136@tcp17 [8/256/0/180] [14318.003646] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14318.045622] LNet: Added LNI 192.168.206.136@tcp18 [8/256/0/180] [14323.578491] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14323.603765] LNet: Added LNI 192.168.206.136@tcp19 [8/256/0/180] [14328.592432] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14328.617866] LNet: Added LNI 192.168.206.136@tcp20 [8/256/0/180] [14334.145957] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14334.249281] LNet: Added LNI 192.168.206.136@tcp21 [8/256/0/180] [14340.319237] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14340.342129] LNet: Added LNI 192.168.206.136@tcp22 [8/256/0/180] [14345.471708] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14345.510678] LNet: Added LNI 192.168.206.136@tcp23 [8/256/0/180] [14350.703831] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14350.749372] LNet: Added LNI 192.168.206.136@tcp24 [8/256/0/180] [14357.146471] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14357.173818] LNet: Added LNI 192.168.206.136@tcp25 [8/256/0/180] [14362.932642] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14362.982691] LNet: Added LNI 192.168.206.136@tcp26 [8/256/0/180] [14368.778968] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14368.810413] LNet: Added LNI 192.168.206.136@tcp27 [8/256/0/180] [14374.923555] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14374.960429] LNet: Added LNI 192.168.206.136@tcp28 [8/256/0/180] [14380.788289] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14380.824126] LNet: Added LNI 192.168.206.136@tcp29 [8/256/0/180] [14386.623550] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14386.650761] LNet: Added LNI 192.168.206.136@tcp30 [8/256/0/180] [14392.956204] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14393.004912] LNet: Added LNI 192.168.206.136@tcp31 [8/256/0/180] [14399.171963] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14399.203285] LNet: Added LNI 192.168.206.136@tcp32 [8/256/0/180] [14405.358503] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14405.405540] LNet: Added LNI 192.168.206.136@tcp33 [8/256/0/180] [14411.689803] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14411.711715] LNet: Added LNI 192.168.206.136@tcp34 [8/256/0/180] [14418.359244] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14418.410161] LNet: Added LNI 192.168.206.136@tcp35 [8/256/0/180] [14425.826257] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14425.857602] LNet: Added LNI 192.168.206.136@tcp36 [8/256/0/180] [14433.566177] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14433.636609] LNet: Added LNI 192.168.206.136@tcp37 [8/256/0/180] [14441.504841] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14441.537527] LNet: Added LNI 192.168.206.136@tcp38 [8/256/0/180] [14449.200082] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14449.224986] LNet: Added LNI 192.168.206.136@tcp39 [8/256/0/180] [14455.797830] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14455.836638] LNet: Added LNI 192.168.206.136@tcp40 [8/256/0/180] [14464.165426] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14464.206943] LNet: Added LNI 192.168.206.136@tcp41 [8/256/0/180] [14470.839126] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14470.862266] LNet: Added LNI 192.168.206.136@tcp42 [8/256/0/180] [14478.678936] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14478.712744] LNet: Added LNI 192.168.206.136@tcp43 [8/256/0/180] [14487.274917] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14487.312979] LNet: Added LNI 192.168.206.136@tcp44 [8/256/0/180] [14495.677246] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14495.698222] LNet: Added LNI 192.168.206.136@tcp45 [8/256/0/180] [14503.937925] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14503.982767] LNet: Added LNI 192.168.206.136@tcp46 [8/256/0/180] [14511.089869] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14511.112454] LNet: Added LNI 192.168.206.136@tcp47 [8/256/0/180] [14517.765234] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14517.792401] LNet: Added LNI 192.168.206.136@tcp48 [8/256/0/180] [14525.124455] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14525.157794] LNet: Added LNI 192.168.206.136@tcp49 [8/256/0/180] [14535.414367] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14535.455872] LNet: Added LNI 192.168.206.136@tcp50 [8/256/0/180] [14558.177663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14558.192302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14558.205568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14560.744260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14560.750387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14560.759891] Lustre: Skipped 1 previous similar message [14560.773863] Lustre: Skipped 2 previous similar messages [14562.328440] Lustre: server umount lustre-MDT0000 complete [14566.390223] LustreError: 473544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882873 with bad export cookie 11617945614924429064 [14566.407809] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14566.411329] LustreError: 473544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14566.865792] Lustre: server umount lustre-MDT0001 complete [14586.336173] Lustre: 472826:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787882877/real 1787882877] req@00000000006892c3 x1874730674195712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787882893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14586.369953] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14587.360101] Lustre: 472825:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787882878/real 1787882878] req@00000000098c7f6a x1874730674195840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787882894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14588.832821] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [14590.497141] Lustre: 472825:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787882881/real 1787882881] req@000000005d485f69 x1874730674195968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787882897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14592.563064] Lustre: 472826:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787882883/real 1787882883] req@00000000e69554e5 x1874730674196224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787882899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14597.728112] Lustre: 472826:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787882888/real 1787882888] req@000000000e052aff x1874730674196352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787882904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14598.364753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14612.337524] LustreError: 475515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14612.361379] LustreError: 475515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14612.408243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14615.459269] LustreError: 475516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14615.468678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14617.037577] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14617.512256] LustreError: 475515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14622.700092] LustreError: 475516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14626.215660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14626.664415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14631.791045] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14631.890541] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14640.271755] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14646.908730] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14667.750149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14667.757553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14667.762483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14667.783033] Lustre: Skipped 1 previous similar message [14672.872388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14672.878325] Lustre: Skipped 2 previous similar messages [14673.147481] Lustre: server umount lustre-OST0000 complete [14677.985628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14677.995820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14678.017097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14678.026318] Lustre: Skipped 1 previous similar message [14685.511830] Lustre: server umount lustre-MDT0000 complete [14688.229208] LustreError: 499768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14688.262292] LustreError: 499768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14688.852915] LustreError: 473546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882995 with bad export cookie 11617945614924431591 [14688.863559] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14688.866874] LustreError: 473546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14689.285087] Lustre: server umount lustre-MDT0001 complete [14701.183283] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [14704.251464] Key type lgssc unregistered [14704.620123] LNet: 502568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14704.632561] LNetError: 502568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14704.638921] LNet: Removed LNI 192.168.206.136@tcp [14704.644500] LNet: Removed LNI 192.168.206.136@tcp10 [14704.647946] LNet: Removed LNI 192.168.206.136@tcp11 [14704.651708] LNet: Removed LNI 192.168.206.136@tcp12 [14704.655348] LNet: Removed LNI 192.168.206.136@tcp13 [14704.657320] LNet: Removed LNI 192.168.206.136@tcp14 [14704.661501] LNet: Removed LNI 192.168.206.136@tcp15 [14704.664676] LNet: Removed LNI 192.168.206.136@tcp16 [14704.678521] LNet: Removed LNI 192.168.206.136@tcp17 [14704.684182] LNet: Removed LNI 192.168.206.136@tcp18 [14704.690171] LNet: Removed LNI 192.168.206.136@tcp19 [14704.693252] LNet: Removed LNI 192.168.206.136@tcp20 [14704.695473] LNet: Removed LNI 192.168.206.136@tcp21 [14704.698512] LNet: Removed LNI 192.168.206.136@tcp22 [14704.700323] LNet: Removed LNI 192.168.206.136@tcp23 [14704.703536] LNet: Removed LNI 192.168.206.136@tcp24 [14704.705862] LNet: Removed LNI 192.168.206.136@tcp25 [14704.708604] LNet: Removed LNI 192.168.206.136@tcp26 [14704.711547] LNet: Removed LNI 192.168.206.136@tcp27 [14704.714634] LNet: Removed LNI 192.168.206.136@tcp28 [14704.717617] LNet: Removed LNI 192.168.206.136@tcp29 [14704.720140] LNet: Removed LNI 192.168.206.136@tcp30 [14704.725179] LNet: Removed LNI 192.168.206.136@tcp31 [14704.728739] LNet: Removed LNI 192.168.206.136@tcp32 [14704.731355] LNet: Removed LNI 192.168.206.136@tcp33 [14704.734538] LNet: Removed LNI 192.168.206.136@tcp34 [14704.738773] LNet: Removed LNI 192.168.206.136@tcp35 [14704.742543] LNet: Removed LNI 192.168.206.136@tcp36 [14704.745821] LNet: Removed LNI 192.168.206.136@tcp37 [14704.748450] LNet: Removed LNI 192.168.206.136@tcp38 [14704.751090] LNet: Removed LNI 192.168.206.136@tcp39 [14704.753858] LNet: Removed LNI 192.168.206.136@tcp40 [14704.756329] LNet: Removed LNI 192.168.206.136@tcp41 [14704.758645] LNet: Removed LNI 192.168.206.136@tcp42 [14704.762392] LNet: Removed LNI 192.168.206.136@tcp43 [14704.772054] LNet: Removed LNI 192.168.206.136@tcp44 [14704.774584] LNet: Removed LNI 192.168.206.136@tcp45 [14704.785029] LNet: Removed LNI 192.168.206.136@tcp46 [14704.790350] LNet: Removed LNI 192.168.206.136@tcp47 [14704.795370] LNet: Removed LNI 192.168.206.136@tcp48 [14704.800161] LNet: Removed LNI 192.168.206.136@tcp49 [14704.815929] LNet: Removed LNI 192.168.206.136@tcp50 [14705.525442] Key type .llcrypt unregistered [14705.528563] Key type ._llcrypt unregistered [14726.083202] Key type ._llcrypt registered [14726.089185] Key type .llcrypt registered [14726.243184] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [14743.290406] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [14744.700591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14744.724445] alg: No test for adler32 (adler32-zlib) [14745.802507] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [14746.118534] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [14747.816608] Key type lgssc registered [14749.475556] Lustre: Echo OBD driver; http://www.lustre.org/ [14786.740804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14786.774653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14788.142748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14788.174202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14788.256732] Lustre: lustre-MDT0000: new disk, initializing [14788.333566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14788.366176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14793.117735] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14803.780345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14803.887992] Lustre: 505765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14803.937577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14803.943400] Lustre: Skipped 1 previous similar message [14804.006854] Lustre: lustre-MDT0001: new disk, initializing [14804.151963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14804.192368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14804.211665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14809.444820] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14819.354164] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14825.507421] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14834.153805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14834.400527] Lustre: *** cfs_fail_loc=90f, val=0*** [14834.495473] Lustre: lustre-OST0000: new disk, initializing [14834.504054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14834.511821] Lustre: 507780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14834.601441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14842.004815] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14842.472207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14842.507763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14842.635515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14852.305836] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14863.336053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14863.336623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14863.354714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14863.360467] Lustre: Skipped 1 previous similar message [14865.983394] Lustre: server umount lustre-OST0000 complete [14871.010417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14871.026116] LustreError: Skipped 1 previous similar message [14871.032971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14871.051451] Lustre: Skipped 1 previous similar message [14871.054881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14873.574549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14873.577596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14873.599559] Lustre: Skipped 1 previous similar message [14878.690416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14883.820225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14883.824681] Lustre: Skipped 2 previous similar messages [14884.321662] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14884.710093] Lustre: server umount lustre-MDT0000 complete [14888.290021] LustreError: 505756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883195 with bad export cookie 5768882176746354136 [14888.296662] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14888.303475] LustreError: 505756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14888.825706] Lustre: server umount lustre-MDT0001 complete [14901.838426] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [14905.140348] Key type lgssc unregistered [14905.533702] LNet: 509693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14905.555523] LNetError: 509693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14906.600419] LNet: Removed LNI 192.168.206.136@tcp [14907.460090] Key type .llcrypt unregistered [14907.465115] Key type ._llcrypt unregistered [14926.738636] Key type ._llcrypt registered [14926.741337] Key type .llcrypt registered [14926.847968] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [14927.842866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14927.880774] alg: No test for adler32 (adler32-zlib) [14928.920321] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [14929.177920] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [14930.848246] Key type lgssc registered [14932.271287] Lustre: Echo OBD driver; http://www.lustre.org/ [14939.874430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14939.891685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14941.284801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14945.308208] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14951.909745] LustreError: 510964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14951.921586] LustreError: 510964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14952.931087] LustreError: 510963:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14954.339254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14954.662819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14959.124596] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14966.240572] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14972.302701] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14979.585829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14980.068992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14986.251275] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [14993.984708] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15008.366309] loop0: detected capacity change from 0 to 204800000 [15008.569279] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15008.587482] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15008.603660] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15015.657847] loop0: detected capacity change from 0 to 204800000 [15015.914841] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15015.931469] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15015.955501] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15021.324502] loop0: detected capacity change from 0 to 204800000 [15027.379953] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15027.500401] Lustre: fs15246-OST0000: new disk, initializing [15027.506366] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15027.514303] Lustre: 514557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15027.620572] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15033.809791] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [15043.731854] loop1: detected capacity change from 0 to 204800000 [15049.547893] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15049.660871] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15049.708717] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15049.781885] Lustre: fs15246-MDT0000: new disk, initializing [15049.921627] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15049.955148] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15050.083367] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15051.193542] Lustre: 515677:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002573af70 x1874731451072384/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787883373 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15055.064441] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [15056.305405] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15056.327804] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15056.367562] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15075.833474] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15075.858412] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15080.937395] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15081.649286] Lustre: server umount fs15246-MDT0000 complete [15086.081064] LustreError: 510958:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15090.533527] Lustre: server umount fs15246-OST0000 complete [15106.018020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15106.025768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15106.037805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15108.499473] Lustre: server umount lustre-OST0000 complete [15113.697965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15113.705771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15113.718246] Lustre: Skipped 1 previous similar message [15113.729447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15113.736583] Lustre: Skipped 1 previous similar message [15116.776981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15119.067469] Lustre: server umount lustre-MDT0000 complete [15121.889203] LustreError: 510964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15121.906544] LustreError: 510964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15123.282716] LustreError: 510944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883430 with bad export cookie 16978183173988200751 [15123.292377] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15123.298235] LustreError: 510944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15123.697263] Lustre: server umount lustre-MDT0001 complete [15136.502636] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [15139.533406] Key type lgssc unregistered [15139.846695] LNet: 518776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15139.862290] LNetError: 518776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15139.886465] LNet: Removed LNI 192.168.206.136@tcp [15140.855447] Key type .llcrypt unregistered [15140.861781] Key type ._llcrypt unregistered [15164.205636] Key type ._llcrypt registered [15164.207306] Key type .llcrypt registered [15164.334941] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [15165.687719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15165.858304] alg: No test for adler32 (adler32-zlib) [15167.048967] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [15167.355379] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [15169.064176] Key type lgssc registered [15170.476991] Lustre: Echo OBD driver; http://www.lustre.org/ [15214.516823] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [15226.692372] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [15268.679789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15268.719617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15270.022598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15270.076531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15270.255740] Lustre: lustre-MDT0000: new disk, initializing [15270.446170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15270.483054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15275.749835] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [15287.086096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15287.232462] Lustre: 523249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15287.340449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15287.348735] Lustre: Skipped 1 previous similar message [15287.424728] Lustre: lustre-MDT0001: new disk, initializing [15287.517743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15287.550554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15287.559337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15292.506704] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [15302.206622] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15308.293259] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15315.985777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15316.187454] Lustre: lustre-OST0000: new disk, initializing [15316.190373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15316.194487] Lustre: 525216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15316.260494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15323.493130] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [15324.236494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15324.250679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15324.382233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15334.103415] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15339.308571] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15339.655235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15344.832330] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15345.019966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15350.241143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15350.242705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15350.248267] LustreError: Skipped 1 previous similar message [15350.248861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15350.272141] Lustre: Skipped 1 previous similar message [15352.954866] Lustre: server umount lustre-OST0000 complete [15359.460814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15359.471725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15359.528035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15359.536316] Lustre: Skipped 1 previous similar message [15360.481333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15363.942350] Lustre: server umount lustre-MDT0000 complete [15365.603116] LustreError: 524846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15365.617920] LustreError: 524846:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [15367.673354] LustreError: 523241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883674 with bad export cookie 11094712170862049556 [15367.697094] LustreError: 523241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15367.698290] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15368.381771] Lustre: server umount lustre-MDT0001 complete [15394.928380] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [15406.359637] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [15443.168546] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15148 sec ========= 22:22:27 (1787883747) [15445.163276] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:22:29 (1787883749) === [15449.795903] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:22:33 (1787883753) === [15497.002861] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [15500.115456] Key type lgssc unregistered [15500.352982] LNet: 531865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15500.366034] LNetError: 531865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15500.388495] LNet: Removed LNI 192.168.206.136@tcp [15501.492461] Key type .llcrypt unregistered [15501.494728] Key type ._llcrypt unregistered