[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495591779 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007434] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021438] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026220] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] *** VALIDATE tmpfs *** [ 0.030462] *** VALIDATE proc *** [ 0.031268] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036038] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040151] debug: unmapping init [mem 0xffffffffaa459000-0xffffffffaa460fff] [ 0.041524] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043017] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.045891] ... value mask: 0000ffffffffffff [ 0.046016] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048014] ... event mask: 000000070000000f [ 0.049319] rcu: Hierarchical SRCU implementation. [ 0.051344] smp: Bringing up secondary CPUs ... [ 0.052686] x86: Booting SMP configuration: [ 0.053032] .... node #0, CPUs: #1 #2 #3 [ 0.056395] smp: Brought up 1 node, 4 CPUs [ 0.058013] smpboot: Max logical packages: 1 [ 0.059017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137027] node 0 deferred pages initialised in 76ms [ 0.141009] devtmpfs: initialized [ 0.142269] x86/mm: Memory block size: 128MB [ 0.144703] gcov: version magic: 0x41383552 [ 0.147303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153340] pinctrl core: initialized pinctrl subsystem [ 0.154187] [ 0.154622] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ** ** [ 0.161013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163015] ** ** [ 0.165014] ** This means that this kernel is built to expose internal ** [ 0.167013] ** IOMMU data structures, which may compromise security on ** [ 0.169012] ** your system. ** [ 0.171012] ** ** [ 0.174014] ** If you see this message and you are not debugging the ** [ 0.176009] ** kernel, report this immediately to your vendor! ** [ 0.178014] ** ** [ 0.180014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182013] ************************************************************* [ 0.184565] NET: Registered protocol family 16 [ 0.185338] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191141] cpuidle: using governor menu [ 0.192922] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193435] PCI: Using configuration type 1 for base access [ 0.194139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206131] cryptd: max_cpu_qlen set to 1000 [ 0.209256] ACPI: Added _OSI(Module Device) [ 0.211016] ACPI: Added _OSI(Processor Device) [ 0.212012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218129] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226299] ACPI: Interpreter enabled [ 0.227000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227000] ACPI: Using IOAPIC for interrupt routing [ 0.229125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232615] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244887] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258393] acpiphp: Slot [2] registered [ 0.260123] acpiphp: Slot [5] registered [ 0.261112] acpiphp: Slot [6] registered [ 0.263119] acpiphp: Slot [7] registered [ 0.264095] acpiphp: Slot [8] registered [ 0.265000] acpiphp: Slot [9] registered [ 0.265000] acpiphp: Slot [10] registered [ 0.267150] acpiphp: Slot [3] registered [ 0.268091] acpiphp: Slot [4] registered [ 0.270100] acpiphp: Slot [11] registered [ 0.271065] acpiphp: Slot [12] registered [ 0.272098] acpiphp: Slot [13] registered [ 0.274123] acpiphp: Slot [14] registered [ 0.275122] acpiphp: Slot [15] registered [ 0.277080] acpiphp: Slot [16] registered [ 0.278083] acpiphp: Slot [17] registered [ 0.279073] acpiphp: Slot [18] registered [ 0.281099] acpiphp: Slot [19] registered [ 0.282116] acpiphp: Slot [20] registered [ 0.283108] acpiphp: Slot [21] registered [ 0.285104] acpiphp: Slot [22] registered [ 0.286092] acpiphp: Slot [23] registered [ 0.288103] acpiphp: Slot [24] registered [ 0.289104] acpiphp: Slot [25] registered [ 0.290094] acpiphp: Slot [26] registered [ 0.292159] acpiphp: Slot [27] registered [ 0.294105] acpiphp: Slot [28] registered [ 0.295074] acpiphp: Slot [29] registered [ 0.296085] acpiphp: Slot [30] registered [ 0.297107] acpiphp: Slot [31] registered [ 0.299115] PCI host bridge to bus 0000:00 [ 0.300016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.312031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317285] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319183] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334506] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345611] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348946] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355630] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373831] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384796] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.420622] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.438021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470523] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.513382] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.520014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.555019] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.565017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.586577] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.593013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.599000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.612015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.622870] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.625363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.627388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.637134] iommu: Default domain type: Passthrough [ 0.639439] SCSI subsystem initialized [ 0.641152] ACPI: bus type USB registered [ 0.643116] usbcore: registered new interface driver usbfs [ 0.644065] usbcore: registered new interface driver hub [ 0.646065] usbcore: registered new device driver usb [ 0.647173] pps_core: LinuxPPS API ver. 1 registered [ 0.649012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.652082] PTP clock support registered [ 0.654151] EDAC MC: Ver: 3.0.0 [ 0.656512] PCI: Using ACPI for IRQ routing [ 0.659893] NetLabel: Initializing [ 0.661010] NetLabel: domain hash size = 128 [ 0.662009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.664074] NetLabel: unlabeled traffic allowed by default [ 0.666111] vgaarb: loaded [ 0.667319] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.668000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.673000] clocksource: Switched to clocksource kvm-clock [ 0.778180] VFS: Disk quotas dquot_6.6.0 [ 0.779662] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782100] *** VALIDATE ramfs *** [ 0.783482] *** VALIDATE hugetlbfs *** [ 0.785801] pnp: PnP ACPI init [ 0.788111] pnp: PnP ACPI: found 6 devices [ 0.808872] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812413] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814742] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817018] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.819422] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.821766] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824467] NET: Registered protocol family 2 [ 0.826937] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.831721] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.835558] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841083] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844972] TCP: Hash tables configured (established 65536 bind 65536) [ 0.848209] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.851790] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854438] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857446] NET: Registered protocol family 1 [ 0.860188] RPC: Registered named UNIX socket transport module. [ 0.862343] RPC: Registered udp transport module. [ 0.864076] RPC: Registered tcp transport module. [ 0.865661] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.867717] NET: Registered protocol family 44 [ 0.869476] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.871656] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.873702] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.876395] PCI: CLS 0 bytes, default 64 [ 0.877882] Unpacking initramfs... [ 2.270587] debug: unmapping init [mem 0xffff9790fcc54000-0xffff9790fffbffff] [ 2.274539] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.276930] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.280031] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.783645] Initialise system trusted keyrings [ 2.785289] Key type blacklist registered [ 2.787995] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.796914] zbud: loaded [ 2.800209] *** VALIDATE nfs *** [ 2.801658] *** VALIDATE nfs4 *** [ 2.803438] pstore: using deflate compression [ 2.808431] Platform Keyring initialized [ 2.937272] NET: Registered protocol family 38 [ 2.939236] Key type asymmetric registered [ 2.941252] Asymmetric key parser 'x509' registered [ 2.943386] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.947276] io scheduler mq-deadline registered [ 2.949162] io scheduler kyber registered [ 2.951412] io scheduler bfq registered [ 2.953561] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.956316] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.959109] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.962157] ACPI: Power Button [PWRF] [ 2.968290] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.975741] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.991771] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.000796] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.019326] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.051503] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.082408] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.089281] Non-volatile memory driver v1.3 [ 3.090827] Linux agpgart interface v0.103 [ 3.128851] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.132186] vda: detected capacity change from 0 to 75194368 [ 3.152716] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.155959] vdb: detected capacity change from 0 to 1073741824 [ 3.170656] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.174076] vdc: detected capacity change from 0 to 2621440000 [ 3.197353] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.200181] vdd: detected capacity change from 0 to 2621440000 [ 3.217509] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.220441] vde: detected capacity change from 0 to 4294967296 [ 3.234873] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238063] vdf: detected capacity change from 0 to 4294967296 [ 3.245925] libphy: Fixed MDIO Bus: probed [ 3.264756] usbcore: registered new interface driver usbserial_generic [ 3.267328] usbserial: USB Serial support registered for generic [ 3.269796] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.273830] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.275633] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.278045] mousedev: PS/2 mouse device common for all mice [ 3.281575] rtc_cmos 00:05: RTC can wake from S4 [ 3.283929] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.286389] rtc_cmos 00:05: registered as rtc0 [ 3.291362] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.291712] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.297772] intel_pstate: CPU model not supported [ 3.301403] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.301487] hid: raw HID events driver (C) Jiri Kosina [ 3.307564] usbcore: registered new interface driver usbhid [ 3.309391] usbhid: USB HID core driver [ 3.310725] drop_monitor: Initializing network drop monitor service [ 3.312810] Initializing XFRM netlink socket [ 3.314697] NET: Registered protocol family 10 [ 3.317481] Segment Routing with IPv6 [ 3.318664] NET: Registered protocol family 17 [ 3.320531] mpls_gso: MPLS GSO support [ 3.325747] RAS: Correctable Errors collector initialized. [ 3.327884] AVX version of gcm_enc/dec engaged. [ 3.329307] AES CTR mode by8 optimization enabled [ 3.425516] sched_clock: Marking stable (3425470173, 0)->(4332466819, -906996646) [ 3.429046] registered taskstats version 1 [ 3.432350] Loading compiled-in X.509 certificates [ 3.434578] zswap: loaded using pool lzo/zbud [ 3.464266] Key type big_key registered [ 3.478068] Key type encrypted registered [ 3.479488] ima: No TPM chip found, activating TPM-bypass! [ 3.481684] ima: Allocated hash algorithm: sha1 [ 3.483453] ima: No architecture policies found [ 3.485400] evm: Initialising EVM extended attributes: [ 3.487654] evm: security.selinux [ 3.489042] evm: security.ima [ 3.490314] evm: security.capability [ 3.491863] evm: HMAC attrs: 0x1 [ 3.494294] rtc_cmos 00:05: setting system clock to 2026-08-27 16:09:37 UTC (1787846977) [ 3.500812] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.504447] debug: unmapping init [mem 0xffffffffaa182000-0xffffffffaa458fff] [ 3.516404] Write protecting the kernel read-only data: 28672k [ 3.519757] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.522168] debug: unmapping init [mem 0xffffffffa9114000-0xffffffffa91fffff] [ 3.558822] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.567549] systemd[1]: Detected virtualization kvm. [ 3.569413] systemd[1]: Detected architecture x86-64. [ 3.571351] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.598793] systemd[1]: No hostname configured. [ 3.600652] systemd[1]: Set hostname to . [ 3.602840] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605409] systemd[1]: Initializing machine ID from random generator. [ 3.660179] random: ln: uninitialized urandom read (6 bytes read) [ 3.757337] random: systemd: uninitialized urandom read (16 bytes read) [ 3.760301] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.763740] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.768729] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.476667] device-mapper: uevent: version 1.0.3 [ 4.478821] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.802032] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.240935] virtio_net virtio0 ens2: renamed from eth0 [ 5.323530] scsi host0: ata_piix [ 5.348441] scsi host1: ata_piix [ 5.349911] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.351949] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.831172] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.996928] random: crng init done [ 9.997954] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.624195] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.824753] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.104726] SELinux: Disabled at runtime. [ 12.168891] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.176488] systemd[1]: Detected virtualization kvm. [ 12.178268] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.677429] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.680492] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.684542] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.687430] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.689888] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.696927] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.700331] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ 12.737957] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.114903] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.343244] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.463988] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.558114] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.583320] EDAC sbridge: Ver: 1.1.2 [ 15.369446] Key type dns_resolver registered [ 15.680198] NFS: Registering the id_resolver key type [ 15.682162] Key type id_resolver registered [ 15.683714] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 73.172254] libcfs: loading out-of-tree module taints kernel. [ 73.237416] Key type ._llcrypt registered [ 73.240179] Key type .llcrypt registered [ 73.346137] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 96.406959] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 98.428546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.447229] alg: No test for adler32 (adler32-zlib) [ 100.139489] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 101.642990] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 103.487231] Key type lgssc registered [ 105.974676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.450692] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 175.458836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 191.367245] hrtimer: interrupt took 2222721 ns [ 194.474262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.541369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.915232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.019611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.207640] Lustre: lustre-MDT0000: new disk, initializing [ 196.382630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.405643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.013740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.147739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.303314] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.342807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.353315] Lustre: Skipped 1 previous similar message [ 217.475802] Lustre: lustre-MDT0001: new disk, initializing [ 217.560921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.600482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.609146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.193590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.911330] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.408849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.634962] Lustre: lustre-OST0000: new disk, initializing [ 236.638537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.643461] Lustre: 8468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.708206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.728946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.736315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.843499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.226963] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.104230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.214826] Lustre: lustre-OST0001: new disk, initializing [ 257.218391] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 257.223423] Lustre: 9542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.295839] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.422176] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.338582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.358441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 267.528417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.287700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.979546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.636244] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.747612] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 302.688870] Lustre: DEBUG MARKER: Client: 2.17.57.87 [ 306.033218] Lustre: DEBUG MARKER: MDS: 2.17.57.87 [ 309.253598] Lustre: DEBUG MARKER: OSS: 2.17.57.87 [ 311.182989] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:14:43 EDT 2026 [ 329.829572] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 331.332645] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.665108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.666054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.681041] Lustre: Skipped 1 previous similar message [ 349.697809] Lustre: Skipped 3 previous similar messages [ 354.723542] Lustre: server umount lustre-MDT0000 complete [ 358.900196] LustreError: 7486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847332 with bad export cookie 15009971589642198373 [ 358.903115] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.923262] LustreError: 7486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 359.905234] LustreError: 11421:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.906129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.913048] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.931949] LustreError: 11421:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 359.947208] Lustre: Skipped 3 previous similar messages [ 363.936381] LustreError: 11421:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.936824] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.992764] Lustre: Skipped 2 previous similar messages [ 365.300795] Lustre: server umount lustre-MDT0001 complete [ 369.706677] Lustre: server umount lustre-OST0000 complete [ 373.620277] Lustre: server umount lustre-OST0001 complete [ 380.514601] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 391.201231] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 428.421061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.677233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.717095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.780509] Lustre: lustre-MDT0000: new disk, initializing [ 428.862208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.885073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.831972] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 446.798971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.991527] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 447.114497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 447.120486] Lustre: Skipped 1 previous similar message [ 447.267185] Lustre: lustre-MDT0001: new disk, initializing [ 447.438378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.511397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.541049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.168262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 465.464990] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.060211] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.797284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.316712] Lustre: lustre-OST0000: new disk, initializing [ 482.320568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.341172] Lustre: 17065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.510723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.691286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 483.695544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 483.961880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.875556] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 500.384639] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 505.551319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 505.769328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 510.980526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.230603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 514.528675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.537135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.539597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.584212] Lustre: Skipped 1 previous similar message [ 514.603940] Lustre: Skipped 1 previous similar message [ 519.650425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.663797] Lustre: Skipped 1 previous similar message [ 519.973910] Lustre: server umount lustre-OST0000 complete [ 524.256981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.276737] LustreError: Skipped 1 previous similar message [ 524.279084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.890392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.901247] Lustre: Skipped 3 previous similar messages [ 530.833593] Lustre: server umount lustre-MDT0000 complete [ 535.013899] LustreError: 15102:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.041205] LustreError: 15102:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 535.158455] LustreError: 15089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847509 with bad export cookie 15009971589642199913 [ 535.161674] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.171937] LustreError: 15089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.549312] Lustre: server umount lustre-MDT0001 complete [ 541.120988] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:18:33 (1787847513) [ 544.143284] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 546.645791] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:18:38 (1787847518) [ 554.546588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.071066] LustreError: 19476:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.136804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.491684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 560.624661] LustreError: 19477:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.246577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.741338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 584.883814] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.587570] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.524242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 605.744332] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 613.746794] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 625.636185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.643216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.643625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.653102] Lustre: Skipped 2 previous similar messages [ 625.672408] Lustre: Skipped 1 previous similar message [ 630.766758] LustreError: 21968:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.793475] LustreError: 21968:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 630.959420] Lustre: server umount lustre-OST0000 complete [ 635.878231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.891208] Lustre: Skipped 1 previous similar message [ 640.578917] Lustre: server umount lustre-MDT0000 complete [ 644.564530] LustreError: 20215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847618 with bad export cookie 15009971589642200788 [ 644.569781] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.577562] LustreError: 20215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 645.176189] Lustre: server umount lustre-MDT0001 complete [ 655.933858] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 658.777777] Key type lgssc unregistered [ 659.072735] LNet: 23013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.083754] LNetError: 23013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.108512] LNet: Removed LNI 192.168.204.146@tcp [ 659.931542] Key type .llcrypt unregistered [ 659.934903] Key type ._llcrypt unregistered [ 681.140401] Key type ._llcrypt registered [ 681.143478] Key type .llcrypt registered [ 681.267350] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 682.193997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.220785] alg: No test for adler32 (adler32-zlib) [ 683.321458] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 683.557876] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 685.215200] Key type lgssc registered [ 686.280210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.347272] loop: module loaded [ 694.401429] loop0: detected capacity change from 0 to 204800000 [ 694.650846] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.664657] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.096725] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.273570] loop0: detected capacity change from 0 to 204800000 [ 701.546620] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.557514] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.578308] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.276904] loop0: detected capacity change from 0 to 204800000 [ 712.799979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 712.812174] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.070129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 714.103977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 714.181464] Lustre: lustre-MDT0000: new disk, initializing [ 714.248176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.272665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 718.484484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 728.913026] loop1: detected capacity change from 0 to 204800000 [ 735.876683] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.315610] Lustre: lustre-OST07c6: new disk, initializing [ 736.323022] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 736.326270] Lustre: Skipped 1 previous similar message [ 736.329386] Lustre: 26216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 736.391364] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 744.174623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 745.153410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 745.163525] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 745.195453] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 749.931343] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 749.938701] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 749.950305] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 749.966165] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 749.982651] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 749.991680] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 766.592496] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 766.619576] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.629367] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.640234] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 766.652367] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 766.661797] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.739297] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 771.750777] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.766404] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 771.774882] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.781326] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.792403] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.806105] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 771.825028] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.852847] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.866419] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.879746] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.898290] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.193815] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 775.199826] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 775.206410] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 775.219269] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.226654] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.242251] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.253297] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 775.259542] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.264204] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.270640] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.276891] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.281500] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.361676] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 780.374714] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 780.384793] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.401019] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.413667] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.445761] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.452591] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 780.460845] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.477588] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 780.489981] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.495047] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.508134] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.807927] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 788.828896] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 788.855996] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 788.890034] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 788.921212] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.945403] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 788.952625] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 788.968783] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 788.983395] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 789.005149] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.015136] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.040522] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 805.582379] Lustre: 26882:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 805.595853] Lustre: 26882:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 805.607280] Lustre: 26882:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.616892] Lustre: 26882:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 805.626725] Lustre: 26882:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.638898] Lustre: 26882:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 805.645845] Lustre: 26882:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 805.653579] Lustre: 26882:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 805.662082] Lustre: 26882:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 805.672824] Lustre: 26882:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 805.681839] Lustre: 26882:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.694705] Lustre: 26882:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 847.846823] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.868592] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.881807] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 849.087373] Lustre: server umount lustre-OST07c6 complete [ 858.724630] Lustre: server umount lustre-MDT0000 complete [ 876.574237] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 879.667842] Key type lgssc unregistered [ 879.959680] LNet: 28775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.978353] LNetError: 28775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.991489] LNet: Removed LNI 192.168.204.146@tcp [ 880.770806] Key type .llcrypt unregistered [ 880.772781] Key type ._llcrypt unregistered [ 901.877860] Key type ._llcrypt registered [ 901.882404] Key type .llcrypt registered [ 902.016428] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 903.710538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.800993] alg: No test for adler32 (adler32-zlib) [ 904.965386] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 905.275165] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 906.952305] Key type lgssc registered [ 908.216842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.357180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.375985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.985417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.548946] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 926.187433] LustreError: 30050:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.215949] LustreError: 30050:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 930.652351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.090159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.767101] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 944.212652] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 950.120974] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 957.043868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.513576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.654821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 963.290191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 971.622708] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 991.719261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.720996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.730431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.739518] Lustre: Skipped 1 previous similar message [ 991.772579] Lustre: Skipped 1 previous similar message [ 996.592488] Lustre: server umount lustre-OST0000 complete [ 1001.957352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.960063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.970977] Lustre: Skipped 1 previous similar message [ 1007.073241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.088413] Lustre: Skipped 2 previous similar messages [ 1011.174996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.185768] Lustre: Skipped 1 previous similar message [ 1015.263628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1015.414336] Lustre: server umount lustre-MDT0000 complete [ 1016.291316] LustreError: 30050:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.319992] LustreError: 30050:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1020.236975] LustreError: 30031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847994 with bad export cookie 5529764337188375430 [ 1020.258860] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.887860] Lustre: server umount lustre-MDT0001 complete [ 1032.847936] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1035.712715] Key type lgssc unregistered [ 1036.059918] LNet: 33790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.067284] LNetError: 33790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.085170] LNet: Removed LNI 192.168.204.146@tcp [ 1036.813610] Key type .llcrypt unregistered [ 1036.818506] Key type ._llcrypt unregistered [ 1056.627448] Key type ._llcrypt registered [ 1056.632240] Key type .llcrypt registered [ 1056.737939] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1057.822600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.836472] alg: No test for adler32 (adler32-zlib) [ 1058.891801] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1059.164730] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1060.849036] Key type lgssc registered [ 1062.351934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.411458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1070.437723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.944103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.881115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1081.317716] LustreError: 35063:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.342312] LustreError: 35063:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1086.060863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.442356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.697637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1099.791522] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.984788] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.635094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.101258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.154498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1118.782233] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1127.340449] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1146.853248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.854111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.856952] LustreError: Skipped 1 previous similar message [ 1146.859786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.863919] Lustre: Skipped 1 previous similar message [ 1146.885173] Lustre: Skipped 1 previous similar message [ 1148.238566] Lustre: server umount lustre-OST0000 complete [ 1152.996500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1153.003274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.020522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.089382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.092177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.136269] Lustre: Skipped 1 previous similar message [ 1158.766712] Lustre: server umount lustre-MDT0000 complete [ 1162.209811] LustreError: 35063:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.240756] LustreError: 35063:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1163.338383] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848137 with bad export cookie 5088716658571149591 [ 1163.342918] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.350984] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1164.036678] Lustre: server umount lustre-MDT0001 complete [ 1176.726155] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1180.284897] Key type lgssc unregistered [ 1180.620713] LNet: 38622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.632867] LNetError: 38622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.670680] LNet: Removed LNI 192.168.204.146@tcp [ 1181.858915] Key type .llcrypt unregistered [ 1181.862418] Key type ._llcrypt unregistered [ 1208.320335] Key type ._llcrypt registered [ 1208.324769] Key type .llcrypt registered [ 1208.453639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1209.655101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1209.916439] alg: No test for adler32 (adler32-zlib) [ 1210.995266] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1211.271756] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1212.983274] Key type lgssc registered [ 1214.181919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.633061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.663518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.172690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.216856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1233.378755] LustreError: 40037:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.403025] LustreError: 40037:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1237.881938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.348452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.904827] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1252.445374] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1258.883353] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1267.835841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.346119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.853478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1275.847654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1284.178782] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.305607] Lustre: server umount lustre-MDT0000 complete [ 1292.399039] LustreError: 40033:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.293084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.309763] Lustre: Skipped 1 previous similar message [ 1296.379644] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848270 with bad export cookie 18138882417822696032 [ 1296.394582] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.471711] LustreError: 40033:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.502037] LustreError: 40033:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1297.517928] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 1298.405941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.417874] Lustre: Skipped 1 previous similar message [ 1298.427580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1298.896212] Lustre: server umount lustre-MDT0001 complete [ 1320.529263] Lustre: server umount lustre-OST0000 complete [ 1339.968311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1342.727226] Key type lgssc unregistered [ 1342.954696] LNet: 43897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.959621] LNetError: 43897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.980385] LNet: Removed LNI 192.168.204.146@tcp [ 1344.834799] Key type .llcrypt unregistered [ 1344.838257] Key type ._llcrypt unregistered [ 1369.074314] Key type ._llcrypt registered [ 1369.077238] Key type .llcrypt registered [ 1369.225807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1370.734816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.750715] alg: No test for adler32 (adler32-zlib) [ 1371.880539] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1372.181773] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1373.871239] Key type lgssc registered [ 1375.016772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1382.880705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1382.894692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.244624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.214657] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1394.671514] LustreError: 45318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.714757] LustreError: 45318:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1396.538837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.957279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.314832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1411.789699] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1419.623991] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1427.317619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.745756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.831496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1434.313679] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1442.318409] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1450.072760] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1450.465173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.469106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.488530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.488540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.499229] Lustre: Skipped 1 previous similar message [ 1451.517353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1455.177197] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1455.182840] Lustre: Skipped 1 previous similar message [ 1455.838419] Lustre: server umount lustre-OST0000 complete [ 1477.089209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.103545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.113336] Lustre: Skipped 1 previous similar message [ 1480.523575] Lustre: server umount lustre-MDT0000 complete [ 1482.208094] LustreError: 46300:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.235516] LustreError: 46300:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1485.096761] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848459 with bad export cookie 9619195222450198023 [ 1485.111426] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1485.113443] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.587064] Lustre: server umount lustre-MDT0001 complete [ 1501.940618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1504.899599] Key type lgssc unregistered [ 1505.193943] LNet: 49029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.205545] LNetError: 49029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.220788] LNet: Removed LNI 192.168.204.146@tcp [ 1506.105824] Key type .llcrypt unregistered [ 1506.109692] Key type ._llcrypt unregistered [ 1529.789069] Key type ._llcrypt registered [ 1529.791064] Key type .llcrypt registered [ 1529.935394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1531.525236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.553714] alg: No test for adler32 (adler32-zlib) [ 1532.585693] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1532.892401] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1534.583189] Key type lgssc registered [ 1535.876160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.945816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.971985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.391354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.874377] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1556.963541] LustreError: 50445:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.989655] LustreError: 50445:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1560.956808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.356118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.568197] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1574.874466] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1581.483662] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1589.161299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1589.618130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.684752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1596.795594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1605.049384] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1611.731316] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1615.144589] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1627.842462] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2712h36m39s [ 1629.566875] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1632.747732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.761091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.765989] Lustre: Skipped 2 previous similar messages [ 1632.778880] Lustre: Skipped 2 previous similar messages [ 1633.383303] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1634.431132] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1634.436562] Lustre: Skipped 3 previous similar messages [ 1638.159345] Lustre: server umount lustre-MDT0000 complete [ 1639.495905] LustreError: 50440:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.878322] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1645.671108] LustreError: 50440:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.712126] LustreError: 50440:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1651.327868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.567598] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.778428] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (not set up) [ 1651.934847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.054335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1652.984703] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1657.895517] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1661.258806] Lustre: DEBUG MARKER: Wait for df (36731) ... [ 1662.890734] Lustre: DEBUG MARKER: done [ 1667.555271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1667.564511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.581039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.580665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.816429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.828469] Lustre: Skipped 5 previous similar messages [ 1681.376542] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1681.829475] Lustre: server umount lustre-OST0000 complete [ 1689.056634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1689.060196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.088634] Lustre: Skipped 1 previous similar message [ 1692.960905] Lustre: server umount lustre-MDT0000 complete [ 1696.226897] LustreError: 51433:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.269501] LustreError: 51433:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 1697.808985] LustreError: 53074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848671 with bad export cookie 2709026014611027291 [ 1697.825792] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.828159] LustreError: 53074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1698.443513] Lustre: server umount lustre-MDT0001 complete [ 1710.897303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1714.151476] Key type lgssc unregistered [ 1714.386752] LNet: 55375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.399475] LNetError: 55375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.431743] LNet: Removed LNI 192.168.204.146@tcp [ 1716.412551] Key type .llcrypt unregistered [ 1716.415268] Key type ._llcrypt unregistered [ 1757.010982] Key type ._llcrypt registered [ 1757.014039] Key type .llcrypt registered [ 1757.138953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1759.085724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.122473] alg: No test for adler32 (adler32-zlib) [ 1760.240864] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1760.486710] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1762.151204] Key type lgssc registered [ 1763.519790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1771.621963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1771.641782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.876655] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1772.896384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1773.014812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.862890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1784.141463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.233312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1784.254877] Lustre: 57589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.484383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1787.624929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1794.884959] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.895454] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.500317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.773413] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1812.093775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.865220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1819.815620] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1827.942987] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1835.079847] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2712h40m07s [ 1837.086735] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1845.907918] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1848.710465] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1848.878887] LustreError: 57597:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1869.420690] Lustre: lustre-MDT0000: Client e76d7bc6-4b11-46ca-9ba7-9c576d707cc4 (at 192.168.204.46@tcp) reconnecting [ 1888.975378] LustreError: 57597:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1891.708963] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1894.772304] Lustre: DEBUG MARKER: done [ 1906.150665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1906.152576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.160188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.184303] Lustre: Skipped 1 previous similar message [ 1906.194301] Lustre: Skipped 1 previous similar message [ 1911.268902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.275983] Lustre: Skipped 1 previous similar message [ 1916.383155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.388996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.406750] Lustre: Skipped 1 previous similar message [ 1916.739426] Lustre: server umount lustre-OST0000 complete [ 1921.506723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.523113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.534296] Lustre: Skipped 1 previous similar message [ 1927.654973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.664111] Lustre: Skipped 3 previous similar messages [ 1935.327755] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1935.615730] Lustre: server umount lustre-MDT0000 complete [ 1939.741071] LustreError: 57581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848913 with bad export cookie 11878603955714865961 [ 1939.763948] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.260398] Lustre: server umount lustre-MDT0001 complete [ 1952.808100] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1955.799504] Key type lgssc unregistered [ 1956.119560] LNet: 61932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.131441] LNetError: 61932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.154168] LNet: Removed LNI 192.168.204.146@tcp [ 1957.020280] Key type .llcrypt unregistered [ 1957.023377] Key type ._llcrypt unregistered [ 2002.541087] Key type ._llcrypt registered [ 2002.546435] Key type .llcrypt registered [ 2002.739637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2003.662542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.675523] alg: No test for adler32 (adler32-zlib) [ 2004.905878] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2005.259370] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2006.961846] Key type lgssc registered [ 2008.195665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2011.600540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2011.630604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.088144] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2013.117250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.422586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.744526] Lustre: Failing over lustre-MDT0000 [ 2015.093974] Lustre: server umount lustre-MDT0000 complete [ 2021.679354] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:43:14 (1787848994) [ 2028.054606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.636142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.932629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2041.533423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.731200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2041.766628] Lustre: 64731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2041.993906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2046.570526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2055.897366] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2062.299292] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2069.485783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.720680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2070.041137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.513835] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2078.831882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2087.375532] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2094.158985] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2103.778682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2103.791097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.821994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.802765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.803265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.634047] Lustre: server umount lustre-MDT0000 complete [ 2111.152408] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849085 with bad export cookie 4317460881440702043 [ 2111.160075] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2111.163512] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.562562] Lustre: server umount lustre-MDT0001 complete [ 2115.192630] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2120.370152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2127.882811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.423248] Lustre: 63654:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849089/real 1787849089] req@000000000437093c x1874695531047296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.438817] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.450343] Lustre: Skipped 1 previous similar message [ 2132.447454] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849090/real 1787849090] req@00000000b559d6e1 x1874695531047424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.610426] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849094/real 1787849094] req@000000004aaf79a3 x1874695531047552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.611858] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3beab3cbf9eb073e [ 2136.642898] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 2136.926849] LustreError: 66581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.950457] LustreError: 66581:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2137.012239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.108434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2141.031499] LustreError: 66582:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.436926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2142.050232] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849100/real 1787849100] req@000000003ebda308 x1874695531047936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.052334] LustreError: 68503:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.081219] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2142.113511] LustreError: 68503:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2147.170746] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849105/real 1787849105] req@0000000040f86584 x1874695531048064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2147.300985] LustreError: 68504:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.336738] LustreError: 68504:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2150.833486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.234558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2155.969470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2156.522023] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2160.633613] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.185588] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.930266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.996299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.003651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.028253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.035873] Lustre: Skipped 2 previous similar messages [ 2177.160667] Lustre: server umount lustre-MDT0000 complete [ 2181.092585] LustreError: 66581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.100544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.400760] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849155 with bad export cookie 4317460881440704318 [ 2181.409983] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.420654] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2181.809362] Lustre: server umount lustre-MDT0001 complete [ 2188.808950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2199.084657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.591363] Lustre: 63654:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849160/real 1787849160] req@000000007ab7a1e1 x1874695531074304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.624869] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.635519] Lustre: Skipped 1 previous similar message [ 2206.760397] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3beab3cbf9eb0ca1 [ 2206.770433] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 2207.086509] LustreError: 66582:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.101048] LustreError: 66582:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2207.211559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.272414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2211.947937] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2221.553125] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2222.296987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.118213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2237.557890] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2244.306528] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.794691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2253.806949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.831465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.839377] Lustre: Skipped 2 previous similar messages [ 2253.863141] Lustre: Skipped 2 previous similar messages [ 2258.913969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.923163] Lustre: Skipped 2 previous similar messages [ 2259.757856] Lustre: server umount lustre-MDT0000 complete [ 2263.939691] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849237 with bad export cookie 4317460881440705697 [ 2263.943376] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.947136] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2264.032505] LustreError: 71684:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.046509] LustreError: 71684:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 2264.221193] Lustre: server umount lustre-MDT0001 complete [ 2271.669111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2275.609654] Lustre: DEBUG MARKER: files compared the same [ 2278.108424] Lustre: server umount lustre-OST0000 complete [ 2297.300389] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2300.338613] Key type lgssc unregistered [ 2300.715959] LNet: 74465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.727378] LNetError: 74465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.754979] LNet: Removed LNI 192.168.204.146@tcp [ 2301.849300] Key type .llcrypt unregistered [ 2301.856237] Key type ._llcrypt unregistered [ 2330.485958] Key type ._llcrypt registered [ 2330.490710] Key type .llcrypt registered [ 2330.606940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2331.875056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.171075] alg: No test for adler32 (adler32-zlib) [ 2333.535950] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2333.842313] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2335.583544] Key type lgssc registered [ 2336.700770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.309782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.343301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.941930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.874720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2356.258172] LustreError: 75886:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.280579] LustreError: 75886:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2361.312933] LustreError: 75885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.628613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.988158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.764908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2376.473797] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.818621] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.517292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.789905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.862074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2398.632843] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2408.528343] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2417.631927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2417.645708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.656900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2421.729736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.730832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2421.774805] Lustre: Skipped 1 previous similar message [ 2423.692657] Lustre: server umount lustre-OST0000 complete [ 2428.383875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2428.399940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.421872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.968253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.968547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.090974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.098985] Lustre: Skipped 2 previous similar messages [ 2442.207867] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2442.340556] Lustre: server umount lustre-MDT0000 complete [ 2446.304117] LustreError: 75886:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.346084] LustreError: 75886:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2447.505490] LustreError: 75866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849421 with bad export cookie 18375801022059343206 [ 2447.524464] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.211762] Lustre: server umount lustre-MDT0001 complete [ 2465.022259] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2468.050668] Key type lgssc unregistered [ 2468.375578] LNet: 79425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.382930] LNetError: 79425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.397754] LNet: Removed LNI 192.168.204.146@tcp [ 2469.669961] Key type .llcrypt unregistered [ 2469.672345] Key type ._llcrypt unregistered [ 2500.384395] Key type ._llcrypt registered [ 2500.386149] Key type .llcrypt registered [ 2500.480482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2501.683621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2501.710343] alg: No test for adler32 (adler32-zlib) [ 2502.727754] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2502.956982] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2504.655750] Key type lgssc registered [ 2506.164310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.337771] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.349608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.868120] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2533.984736] LustreError: 80818:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2560.928734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.827655] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2578.265460] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2585.730362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.829845] Lustre: *** cfs_fail_loc=706, val=0*** [ 2586.425269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.579023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2591.429421] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2591.740044] LustreError: 80843:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.771916] LustreError: 80843:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2595.823254] LustreError: 80844:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.869520] LustreError: 80844:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2600.933720] LustreError: 81904:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.953878] LustreError: 81904:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2601.247070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.555301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.230851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2612.101365] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.074121] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2621.928316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2621.928935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.937322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.958613] Lustre: Skipped 1 previous similar message [ 2626.295543] Lustre: server umount lustre-OST0000 complete [ 2632.168906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2632.175475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.191023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.194275] Lustre: Skipped 2 previous similar messages [ 2637.292098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.297711] Lustre: Skipped 1 previous similar message [ 2637.577078] Lustre: server umount lustre-MDT0000 complete [ 2641.278052] LustreError: 82645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849615 with bad export cookie 4340392099237044671 [ 2641.280159] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.292162] LustreError: 82645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2641.686794] Lustre: server umount lustre-MDT0001 complete [ 2653.386623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2656.550643] Key type lgssc unregistered [ 2656.881263] LNet: 84437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.897341] LNetError: 84437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.914260] LNet: Removed LNI 192.168.204.146@tcp [ 2657.934459] Key type .llcrypt unregistered [ 2657.937081] Key type ._llcrypt unregistered [ 2684.533062] Key type ._llcrypt registered [ 2684.537513] Key type .llcrypt registered [ 2684.662939] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2685.914158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2686.048539] alg: No test for adler32 (adler32-zlib) [ 2687.096514] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2687.369758] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2689.015317] Key type lgssc registered [ 2690.352579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.926066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2698.945034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.190149] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2714.375991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.928843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2719.972312] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2720.228884] LustreError: 86542:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.249069] LustreError: 86542:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2724.330266] LustreError: 86541:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.351515] LustreError: 86541:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2729.100654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.405783] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.426866] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2729.452641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.571068] LustreError: 86545:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.589759] LustreError: 86545:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2736.992382] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2739.686393] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.749640] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2746.943290] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.956798] Lustre: MGS: Client 2ca8ccc7-ba31-4345-90c9-18c69b470bcc (at 0@lo) reconnecting [ 2746.984028] LustreError: 86556:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2747.285246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.371929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2751.983822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2767.846783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.847486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2767.857613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.965378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.972675] Lustre: Skipped 1 previous similar message [ 2774.052718] Lustre: server umount lustre-OST0000 complete [ 2783.205691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2783.210611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.240346] Lustre: Skipped 1 previous similar message [ 2783.253138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2783.260019] Lustre: Skipped 1 previous similar message [ 2784.465107] Lustre: server umount lustre-MDT0000 complete [ 2788.322348] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.339885] LustreError: 86546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2788.948394] Lustre: server umount lustre-MDT0001 complete [ 2793.198387] Lustre: server umount MGS complete [ 2809.299190] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:56:21 (1787849781) [ 2841.750158] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2853.644453] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2896.256065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.775683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2912.384178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.634451] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2912.673907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2912.772870] Lustre: lustre-MDT0001: new disk, initializing [ 2912.851952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.904735] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2912.920992] Lustre: 93733:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2912.926924] LustreError: 93733:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2918.425430] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2929.209539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.541456] Lustre: lustre-OST0000: new disk, initializing [ 2929.547616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2929.556268] Lustre: 94499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2929.621993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.590560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2938.464423] Lustre: 94893:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000160c574c x1874696246175104/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787849928 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2947.816204] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2955.850800] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2955.859646] Lustre: MGS: Client 994af925-6bfa-4b56-a84f-a4cca0ca4cce (at 0@lo) reconnecting [ 2955.909152] Lustre: 95544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.923703] LustreError: 93719:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2955.939057] LustreError: 93719:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2955.955744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2956.113986] Lustre: lustre-MDT0000: new disk, initializing [ 2956.363796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.406155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2956.460525] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2957.546872] Lustre: 95568:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003e058371 x1874696246182272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787849947 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2960.975209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2962.655837] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2965.990996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2966.006476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2966.631770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2966.707316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2991.584573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2991.589878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.601908] Lustre: Skipped 1 previous similar message [ 2991.604821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2996.649682] Lustre: server umount lustre-OST0000 complete [ 3003.877111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.909563] Lustre: Skipped 1 previous similar message [ 3003.914828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3003.929356] Lustre: Skipped 4 previous similar messages [ 3007.455341] Lustre: server umount lustre-MDT0000 complete [ 3008.994877] LustreError: 93709:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.025143] LustreError: 93709:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3026.401643] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3026.646343] Lustre: server umount lustre-MDT0001 complete [ 3030.676678] Lustre: server umount MGS complete [ 3042.243139] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:00:14 (1787850014) [ 3059.934721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3063.165660] Key type lgssc unregistered [ 3063.594359] LNet: 98076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3063.611495] LNetError: 98076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.625464] LNet: Removed LNI 192.168.204.146@tcp [ 3064.865758] Key type .llcrypt unregistered [ 3064.872812] Key type ._llcrypt unregistered [ 3082.791583] Key type ._llcrypt registered [ 3082.795732] Key type .llcrypt registered [ 3082.924379] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3084.284840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3084.369225] alg: No test for adler32 (adler32-zlib) [ 3085.375734] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3085.572693] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3087.224430] Key type lgssc registered [ 3088.763972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3096.465371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3096.484329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.869389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.530721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3107.303435] LustreError: 99313:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.319204] LustreError: 99313:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3111.925599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.264836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3117.458305] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3125.217324] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3131.095893] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3138.659497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.093940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.199282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3146.196698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3154.903757] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3168.740397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3168.760415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.775821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.783032] Lustre: Skipped 1 previous similar message [ 3168.796718] Lustre: Skipped 1 previous similar message [ 3173.867699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3173.882708] Lustre: Skipped 1 previous similar message [ 3174.568899] Lustre: server umount lustre-OST0000 complete [ 3178.985178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.988406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3178.995966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3179.008762] Lustre: Skipped 1 previous similar message [ 3184.100420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.113663] Lustre: Skipped 2 previous similar messages [ 3189.216363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.224946] Lustre: Skipped 1 previous similar message [ 3193.312570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3193.702508] Lustre: server umount lustre-MDT0000 complete [ 3194.339866] LustreError: 99312:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.366327] LustreError: 99312:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3197.886507] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850171 with bad export cookie 11378397307658717383 [ 3197.892739] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.896795] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3198.403840] Lustre: server umount lustre-MDT0001 complete [ 3211.875452] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3215.802440] Key type lgssc unregistered [ 3216.331689] LNet: 102871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3216.340694] LNetError: 102871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3216.366128] LNet: Removed LNI 192.168.204.146@tcp [ 3217.172353] Key type .llcrypt unregistered [ 3217.176157] Key type ._llcrypt unregistered [ 3237.457935] Key type ._llcrypt registered [ 3237.461551] Key type .llcrypt registered [ 3237.674639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3238.828549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3238.874988] alg: No test for adler32 (adler32-zlib) [ 3240.091743] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3240.519662] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3242.280273] Key type lgssc registered [ 3243.977797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.134417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.141966] LustreError: 103938:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3252.338306] LustreError: 103957:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3252.356048] LustreError: 103938:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4797 [ 3252.363631] LustreError: 103938:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3252.376294] LustreError: 103938:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3252.401200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.201881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.481951] LustreError: 103994:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.506816] LustreError: 103994:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3266.061074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.256814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.503509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3278.189521] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lsmod [ 3281.455752] LustreError: 105158:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3281.569504] LustreError: 105165:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3281.577410] LustreError: 105158:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4897 [ 3281.589704] LustreError: 105158:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3281.596937] LustreError: 105158:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3281.611869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.004813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.013532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.019936] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.024645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.060717] Lustre: Skipped 2 previous similar messages [ 3290.808581] Lustre: server umount lustre-MDT0000 complete [ 3291.178976] LustreError: 103995:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.214513] LustreError: 103995:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3294.517046] LustreError: 103972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850268 with bad export cookie 16444483237487742454 [ 3294.525506] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.527168] LustreError: 103972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3295.327695] LustreError: 104426:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7b71948 x1874696826015232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3295.528310] Lustre: server umount lustre-MDT0001 complete [ 3310.332601] Lustre: server umount lustre-OST0000 complete [ 3320.919529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.576916] LustreError: 106229:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.708842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.615901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3326.958951] LustreError: 106230:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.042612] LustreError: 106229:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.636557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.974491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.749573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3350.592239] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.069108] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3366.721405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.341190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.674136] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3383.760528] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3391.926296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3402.208747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3402.213749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.228045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3402.238793] Lustre: Skipped 1 previous similar message [ 3402.252728] Lustre: Skipped 3 previous similar messages [ 3407.330483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3407.349203] Lustre: Skipped 1 previous similar message [ 3408.115469] Lustre: server umount lustre-OST0000 complete [ 3417.060766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3417.066279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.081634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.050573] Lustre: server umount lustre-MDT0000 complete [ 3422.689744] LustreError: 106229:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.704290] LustreError: 106229:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3423.518128] LustreError: 108074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850397 with bad export cookie 16444483237487743154 [ 3423.527990] LustreError: 108074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3423.531114] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3423.898790] Lustre: server umount lustre-MDT0001 complete [ 3435.853829] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3438.984240] Key type lgssc unregistered [ 3439.294212] LNet: 109793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3439.324207] LNetError: 109793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3439.345333] LNet: Removed LNI 192.168.204.146@tcp [ 3440.204775] Key type .llcrypt unregistered [ 3440.206527] Key type ._llcrypt unregistered [ 3466.638814] Key type ._llcrypt registered [ 3466.640704] Key type .llcrypt registered [ 3466.790441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3468.231970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3468.275134] alg: No test for adler32 (adler32-zlib) [ 3469.489383] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3469.848586] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3471.608158] Key type lgssc registered [ 3473.120347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.176284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3482.194050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.754213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.866356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3493.348441] LustreError: 111208:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.370141] LustreError: 111208:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3498.465486] LustreError: 111209:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.554703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.007438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3504.356792] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3512.921076] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3520.543787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3528.270607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.654955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.663384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3534.719071] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3543.089817] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3549.523912] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3554.905648] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3560.418268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3560.428737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.443519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.529305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.550962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.558927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.530300] Lustre: server umount lustre-OST0000 complete [ 3574.753711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.776507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3574.790202] Lustre: Skipped 2 previous similar messages [ 3578.028933] Lustre: server umount lustre-MDT0000 complete [ 3579.888332] LustreError: 112163:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.923434] LustreError: 112163:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3582.075380] LustreError: 111942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850556 with bad export cookie 17359962219149821939 [ 3582.081210] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3582.088328] LustreError: 111942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3582.464538] Lustre: server umount lustre-MDT0001 complete [ 3593.556926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3596.834909] Key type lgssc unregistered [ 3597.110996] LNet: 114850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3597.129653] LNetError: 114850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3597.147191] LNet: Removed LNI 192.168.204.146@tcp [ 3598.187540] Key type .llcrypt unregistered [ 3598.194956] Key type ._llcrypt unregistered [ 3616.039944] Key type ._llcrypt registered [ 3616.041920] Key type .llcrypt registered [ 3616.157349] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3617.165067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.192133] alg: No test for adler32 (adler32-zlib) [ 3618.545843] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3618.863407] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3620.575190] Key type lgssc registered [ 3621.926196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.515674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3630.553183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.072115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.792590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3642.339969] LustreError: 116073:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.373662] LustreError: 116073:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3645.729151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.230481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3650.906854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3658.600458] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3665.681061] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.404801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.807224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.947400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3680.517738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3688.763557] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3703.785942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.798565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.815421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.855159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.855790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.876378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.902722] Lustre: server umount lustre-OST0000 complete [ 3717.113342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.141836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.148592] Lustre: Skipped 2 previous similar messages [ 3720.092520] Lustre: server umount lustre-MDT0000 complete [ 3722.213521] LustreError: 116072:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.236056] LustreError: 116072:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3724.815937] LustreError: 116053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850698 with bad export cookie 9161967171873555480 [ 3724.829229] LustreError: 116053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3724.830091] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3725.266392] Lustre: server umount lustre-MDT0001 complete [ 3737.791437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3741.347324] Key type lgssc unregistered [ 3741.732918] LNet: 119630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.745886] LNetError: 119630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.774068] LNet: Removed LNI 192.168.204.146@tcp [ 3742.788363] Key type .llcrypt unregistered [ 3742.796373] Key type ._llcrypt unregistered [ 3776.231226] Key type ._llcrypt registered [ 3776.233325] Key type .llcrypt registered [ 3776.363216] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3778.230876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3778.595575] alg: No test for adler32 (adler32-zlib) [ 3779.818454] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3780.207412] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3782.015180] Key type lgssc registered [ 3783.308347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3793.132576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3793.153540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.717774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.891856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3804.134985] LustreError: 121048:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.160351] LustreError: 121048:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3809.256880] LustreError: 121049:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.120526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.438430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3815.324730] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3823.419914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.069498] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3839.202377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.668518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.733956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3845.861433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3857.704953] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.012715] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3874.465457] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3874.479259] Lustre: Skipped 2 previous similar messages [ 3875.593639] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3875.649579] Lustre: lustre-MDT0000: nosquash_nids is cleared [ 3880.907499] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3880.921265] Lustre: Skipped 2 previous similar messages [ 3883.472948] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3883.487580] Lustre: Skipped 1 previous similar message [ 3894.589453] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3894.601264] Lustre: Skipped 2 previous similar messages [ 3901.109546] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp [ 3906.435682] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.46/24@tcp in log lustre-MDT0000 [ 3906.441204] Lustre: Skipped 2 previous similar messages [ 3909.194044] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 3909.201429] Lustre: Skipped 1 previous similar message [ 3915.761240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3915.763873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.773579] LustreError: Skipped 1 previous similar message [ 3915.799483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3916.769174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3916.770772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.343394] Lustre: server umount lustre-OST0000 complete [ 3921.888187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.894901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.908957] Lustre: Skipped 1 previous similar message [ 3921.920488] Lustre: Skipped 2 previous similar messages [ 3927.013067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.027020] Lustre: Skipped 1 previous similar message [ 3932.129202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.141793] Lustre: Skipped 1 previous similar message [ 3936.223224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3936.423718] Lustre: server umount lustre-MDT0000 complete [ 3940.419046] LustreError: 121029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850914 with bad export cookie 16175769040718811690 [ 3940.425830] LustreError: 121029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3940.426309] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.678816] Lustre: server umount lustre-MDT0001 complete [ 3954.323601] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3957.793332] Key type lgssc unregistered [ 3958.155637] LNet: 125857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3958.170534] LNetError: 125857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3958.217906] LNet: Removed LNI 192.168.204.146@tcp [ 3959.275372] Key type .llcrypt unregistered [ 3959.287493] Key type ._llcrypt unregistered [ 3989.250942] Key type ._llcrypt registered [ 3989.257987] Key type .llcrypt registered [ 3989.455331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3990.558314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.609105] alg: No test for adler32 (adler32-zlib) [ 3991.704860] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3991.948478] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3993.671255] Key type lgssc registered [ 3994.629591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4003.193280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4003.218514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.674607] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4004.697309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.716517] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.46/24@tcp [ 4008.884319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4014.059950] LustreError: 127420:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.083520] LustreError: 127420:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4018.180706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.489496] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4018.536269] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 4018.580181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.709590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4032.225776] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4039.862810] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4047.768354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.110028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.175859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4056.865859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4065.975243] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4079.086046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.088110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.089037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.100931] Lustre: Skipped 1 previous similar message [ 4083.339951] Lustre: server umount lustre-OST0000 complete [ 4089.313702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.317736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.341450] Lustre: Skipped 1 previous similar message [ 4089.354513] Lustre: Skipped 2 previous similar messages [ 4094.313335] Lustre: server umount lustre-MDT0000 complete [ 4094.438323] LustreError: 127416:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.468071] LustreError: 127416:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4098.449880] LustreError: 127399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851072 with bad export cookie 10649462956127726225 [ 4098.459792] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4098.466309] LustreError: 127399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4099.101340] Lustre: server umount lustre-MDT0001 complete [ 4110.593748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4113.140412] Key type lgssc unregistered [ 4113.410259] LNet: 131015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4113.421321] LNetError: 131015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4113.441256] LNet: Removed LNI 192.168.204.146@tcp [ 4114.019895] Key type .llcrypt unregistered [ 4114.022724] Key type ._llcrypt unregistered [ 4146.321644] Key type ._llcrypt registered [ 4146.327811] Key type .llcrypt registered [ 4146.442871] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4164.881477] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4166.275827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4166.394430] alg: No test for adler32 (adler32-zlib) [ 4167.774874] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 4167.937739] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4169.687380] Key type lgssc registered [ 4171.163547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4215.760992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4215.830768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.233495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4217.257991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4217.322944] Lustre: lustre-MDT0000: new disk, initializing [ 4217.375692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.391375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4223.382721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4236.175159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.261815] Lustre: 134416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.363808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4236.381947] Lustre: Skipped 1 previous similar message [ 4236.552652] Lustre: lustre-MDT0001: new disk, initializing [ 4236.794656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.888686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4236.913880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4241.797545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4252.523802] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4260.124905] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4267.665744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4267.923728] Lustre: lustre-OST0000: new disk, initializing [ 4267.928371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4267.939216] Lustre: 136385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4268.047178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.392180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4273.448573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4273.615706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4276.071092] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4288.173148] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4292.983607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4293.293465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4298.006846] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4298.274364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4302.319907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.327232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4302.340441] Lustre: Skipped 1 previous similar message [ 4302.372197] Lustre: Skipped 1 previous similar message [ 4307.430563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.548383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.564055] Lustre: Skipped 2 previous similar messages [ 4315.103957] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4315.529852] Lustre: server umount lustre-OST0000 complete [ 4322.786184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.789537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.828223] Lustre: Skipped 1 previous similar message [ 4327.626313] Lustre: server umount lustre-MDT0000 complete [ 4332.291014] LustreError: 134406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851306 with bad export cookie 10087579449216439223 [ 4332.306614] LustreError: 134406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4332.309329] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.788340] Lustre: server umount lustre-MDT0001 complete [ 4344.322560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.808991] LustreError: 138604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.844000] LustreError: 138604:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4344.923724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.922906] LustreError: 138605:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.268492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4354.019695] LustreError: 138604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.140565] LustreError: 138605:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.902670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.410272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4365.570396] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4376.306626] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4384.506946] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4393.402046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4393.863355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.384112] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4413.726380] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4420.281679] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4420.618468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4427.377768] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4427.683199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4437.915229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.097251] Lustre: lustre-OST0001: new disk, initializing [ 4438.104810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4438.117566] Lustre: 141981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4438.238412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4444.946798] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4446.357825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4446.368086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4446.522919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4455.511633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4455.800259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4461.714571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4462.130991] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4469.853748] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f36029c5000.ost_server_uuid 50 [ 4472.001964] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f36029c5000.ost_server_uuid in IDLE state after 0 sec [ 4485.280162] Lustre: server umount lustre-OST0001 complete [ 4487.664212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.673589] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.688590] LustreError: Skipped 1 previous similar message [ 4487.698649] LustreError: 141098:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.706377] Lustre: Skipped 1 previous similar message [ 4491.233489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.244691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.258483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.266231] Lustre: Skipped 1 previous similar message [ 4496.494293] Lustre: server umount lustre-OST0000 complete [ 4503.009275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.016597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4503.028789] Lustre: Skipped 2 previous similar messages [ 4503.047773] Lustre: Skipped 3 previous similar messages [ 4507.418559] Lustre: server umount lustre-MDT0000 complete [ 4508.134506] LustreError: 140254:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.164664] LustreError: 140254:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4511.566080] LustreError: 138584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851485 with bad export cookie 10087579449216440105 [ 4511.575233] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.577024] LustreError: 138584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4511.903836] Lustre: server umount lustre-MDT0001 complete [ 4533.830047] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4536.479846] Key type lgssc unregistered [ 4536.791080] LNet: 144989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.805757] LNetError: 144989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.851226] LNet: Removed LNI 192.168.204.146@tcp [ 4537.757389] Key type .llcrypt unregistered [ 4537.759517] Key type ._llcrypt unregistered [ 4581.404977] Key type ._llcrypt registered [ 4581.410894] Key type .llcrypt registered [ 4581.583556] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4598.752484] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4600.148777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4600.287226] alg: No test for adler32 (adler32-zlib) [ 4601.428167] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 4601.703489] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4603.407852] Key type lgssc registered [ 4604.863894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4644.925256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4644.982685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.183523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4646.233967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4646.294318] Lustre: lustre-MDT0000: new disk, initializing [ 4646.385985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4646.405296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4651.964599] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4667.517633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.966760] Lustre: 149145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4668.070113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4668.072175] Lustre: Skipped 1 previous similar message [ 4668.159233] Lustre: lustre-MDT0001: new disk, initializing [ 4668.267599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.321718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4668.344691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4674.285961] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4685.869711] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4692.690856] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4699.751701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4700.067412] Lustre: lustre-OST0000: new disk, initializing [ 4700.077514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4700.082891] Lustre: 151118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4700.158991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.068096] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4708.464715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4708.477513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4708.648044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4723.201551] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4729.620491] Lustre: Failing over lustre-OST0000 [ 4730.012704] Lustre: server umount lustre-OST0000 complete [ 4746.213142] Lustre: 147367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851704/real 1787851704] req@000000003182f7ca x1874698253144448/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787851720 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4746.246271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.303171] Lustre: 147370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851708/real 1787851708] req@00000000e90bcbfc x1874698253145600/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4750.345205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.675420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.952296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.966501] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4755.039115] Lustre: 147369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851713/real 1787851713] req@0000000038fb5c4d x1874698253146752/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4755.078484] Lustre: 147369:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4756.076355] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4756.662209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4760.095144] Lustre: 147369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851718/real 1787851718] req@000000005a531e9f x1874698253147776/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4760.130176] Lustre: 147369:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4760.254839] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4760.255266] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4762.350533] Lustre: Failing over lustre-MDT0000 [ 4762.811715] Lustre: server umount lustre-MDT0000 complete [ 4765.666569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4765.680455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.348394] LustreError: 149152:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.385807] LustreError: 149152:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4771.810425] LustreError: 149151:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.843224] LustreError: 149151:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4773.454834] LustreError: 149152:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.929334] LustreError: 150309:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.967662] LustreError: 150309:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4782.048231] Lustre: 147367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851740/real 1787851740] req@00000000f3ea1561 x1874698253160064/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851756 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4782.050499] LustreError: 149156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.111561] Lustre: 147367:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4782.111666] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.148159] LustreError: 149156:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4785.183129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.294470] LustreError: 149152:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.300795] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83f84ba85e7c4cb9 [ 4792.309432] LustreError: 149152:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 4792.336602] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 4792.339413] Lustre: Skipped 1 previous similar message [ 4792.578222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.626311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4794.964619] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4796.732931] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4797.929373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4797.957076] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4797.993095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4806.650190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4806.661132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.675918] Lustre: Skipped 2 previous similar messages [ 4806.685727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.168222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.173363] Lustre: Skipped 1 previous similar message [ 4812.193546] Lustre: server umount lustre-OST0000 complete [ 4818.402543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.412566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.413162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4818.419723] Lustre: Skipped 2 previous similar messages [ 4818.442423] Lustre: Skipped 1 previous similar message [ 4823.356922] Lustre: server umount lustre-MDT0000 complete [ 4823.521922] LustreError: 149153:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.549676] LustreError: 149153:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4827.925936] LustreError: 149138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851801 with bad export cookie 9509433799704595641 [ 4827.955788] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.957685] LustreError: 149138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4828.265067] Lustre: server umount lustre-MDT0001 complete [ 4840.784751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4843.483226] Key type lgssc unregistered [ 4843.707739] LNet: 154777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4843.714219] LNetError: 154777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4843.729251] LNet: Removed LNI 192.168.204.146@tcp [ 4844.595966] Key type .llcrypt unregistered [ 4844.599770] Key type ._llcrypt unregistered [ 4886.283449] Key type ._llcrypt registered [ 4886.287965] Key type .llcrypt registered [ 4886.472655] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4888.331059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4888.375479] alg: No test for adler32 (adler32-zlib) [ 4889.554648] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 4889.924210] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4891.663185] Key type lgssc registered [ 4893.281595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4903.150281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4903.178367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.695609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.376293] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4914.147660] LustreError: 156297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4914.178468] LustreError: 156297:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4919.268483] LustreError: 156296:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.490578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4924.389387] LustreError: 156297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4927.029627] LustreError: 156325:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4927.045870] LustreError: 156325:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4927.056492] LustreError: 156325:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4927.479568] Lustre: server umount lustre-MDT0000 complete [ 4945.984918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4957.382519] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5012.874703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.142936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5013.185315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5013.306881] Lustre: lustre-MDT0000: new disk, initializing [ 5013.481573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.504493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5018.791638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5032.398528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.625557] Lustre: 159635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5032.728521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5032.732739] Lustre: Skipped 1 previous similar message [ 5032.827288] Lustre: lustre-MDT0001: new disk, initializing [ 5033.022654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.079562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5033.110721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5039.292794] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5050.155727] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5056.964776] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5065.063522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5065.365321] Lustre: lustre-OST0000: new disk, initializing [ 5065.371744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5065.381949] Lustre: 161604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5065.537363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.961318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5072.973476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5073.119054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5073.302640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5084.309580] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5089.704257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5091.892957] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:34:24 (1787852064) [ 5421.537622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.540636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.574262] Lustre: Skipped 1 previous similar message [ 5421.593108] Lustre: Skipped 2 previous similar messages [ 5426.676533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.695378] Lustre: Skipped 2 previous similar messages [ 5431.778591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5431.788470] Lustre: Skipped 2 previous similar messages [ 5433.311232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5433.641422] Lustre: server umount lustre-MDT0000 complete [ 5436.899535] LustreError: 159646:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.918776] LustreError: 159646:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5437.868379] LustreError: 159626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852411 with bad export cookie 3298559877468105852 [ 5437.869049] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.883085] LustreError: 159626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5442.018075] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.020251] LustreError: 159640:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.049719] Lustre: Skipped 1 previous similar message [ 5442.070479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5447.140361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5452.256821] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5452.980747] Lustre: server umount lustre-MDT0001 complete [ 5466.634596] Lustre: server umount lustre-OST0000 complete [ 5480.273636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5492.032724] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5538.780239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.011442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5539.067799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5539.150855] Lustre: lustre-MDT0000: new disk, initializing [ 5539.304330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.318598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5544.378703] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5557.065515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.231397] Lustre: 166297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5557.306796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5557.310594] Lustre: Skipped 1 previous similar message [ 5557.385308] Lustre: lustre-MDT0001: new disk, initializing [ 5557.484543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5557.538322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5557.559889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.567549] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5574.467722] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5581.253962] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5590.126690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.478720] Lustre: lustre-OST0000: new disk, initializing [ 5590.483755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5590.491815] Lustre: 168265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5590.619387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5597.307064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5597.325723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5597.474834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5598.828543] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5611.346354] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5617.518501] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5617.864810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5625.523170] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5626.058400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5633.507724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5633.516325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.545826] Lustre: Skipped 1 previous similar message [ 5633.554913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.582165] Lustre: Skipped 2 previous similar messages [ 5637.066796] Lustre: server umount lustre-OST0000 complete [ 5644.650869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.681604] Lustre: Skipped 1 previous similar message [ 5649.890845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5649.916262] Lustre: Skipped 3 previous similar messages [ 5650.984816] Lustre: server umount lustre-MDT0000 complete [ 5653.985872] LustreError: 166306:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.007839] LustreError: 166306:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5655.156333] LustreError: 166288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852629 with bad export cookie 3298559877468264682 [ 5655.160070] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.174231] LustreError: 166288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5655.506626] Lustre: server umount lustre-MDT0001 complete [ 5665.517668] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:43:57 (1787852637) [ 5687.849667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5699.219842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5751.033495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.702835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5751.778624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5752.036590] Lustre: lustre-MDT0000: new disk, initializing [ 5752.375714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.428368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5761.228105] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5774.479663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.646159] Lustre: 173028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5774.908380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5780.291177] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5791.812964] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5800.759534] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5810.617934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.096807] Lustre: lustre-OST0000: new disk, initializing [ 5811.101287] Lustre: Skipped 1 previous similar message [ 5811.114863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5811.120869] Lustre: Skipped 2 previous similar messages [ 5811.131943] Lustre: 175014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5817.516759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5817.527381] Lustre: Skipped 1 previous similar message [ 5817.544570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5817.744141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5822.061715] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5836.579789] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5848.547456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5848.548430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.563276] LustreError: Skipped 2 previous similar messages [ 5848.569394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5848.586985] Lustre: Skipped 1 previous similar message [ 5848.607307] Lustre: Skipped 1 previous similar message [ 5851.216665] Lustre: server umount lustre-OST0000 complete [ 5856.738088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5856.758725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.976780] Lustre: server umount lustre-MDT0000 complete [ 5863.908048] LustreError: 173038:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.942752] LustreError: 173038:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5870.063056] LustreError: 173019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852844 with bad export cookie 3298559877468265564 [ 5870.068742] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.072164] LustreError: 173019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5871.297467] Lustre: server umount lustre-MDT0001 complete [ 5882.858312] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:47:35 (1787852855) [ 5903.158642] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5914.959682] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5964.257167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.699420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5964.745799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5964.857663] Lustre: lustre-MDT0000: new disk, initializing [ 5964.965536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.974542] Lustre: Skipped 2 previous similar messages [ 5965.012444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5970.644187] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5988.136342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.295885] Lustre: 179340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5988.760794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5995.536496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6007.350635] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6013.776363] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6022.747466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.175806] Lustre: 181319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6029.438741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6029.662724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6031.528383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6044.016768] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.398896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.399873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.406840] LustreError: Skipped 1 previous similar message [ 6055.409938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.428103] Lustre: Skipped 2 previous similar messages [ 6055.452465] Lustre: Skipped 4 previous similar messages [ 6059.451221] Lustre: server umount lustre-OST0000 complete [ 6065.636931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6070.532295] Lustre: server umount lustre-MDT0000 complete [ 6070.757176] LustreError: 179351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.787559] LustreError: 179351:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6075.152170] LustreError: 179331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853049 with bad export cookie 3298559877468266649 [ 6075.164642] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.168941] LustreError: 179331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6075.710363] Lustre: server umount lustre-MDT0001 complete [ 6088.396201] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6091.498801] Key type lgssc unregistered [ 6091.791796] LNet: 183181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6091.804510] LNetError: 183181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6091.826157] LNet: Removed LNI 192.168.204.146@tcp [ 6092.964837] Key type .llcrypt unregistered [ 6092.973463] Key type ._llcrypt unregistered [ 6126.468448] Key type ._llcrypt registered [ 6126.472986] Key type .llcrypt registered [ 6126.593305] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6127.475739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6127.487515] alg: No test for adler32 (adler32-zlib) [ 6128.666060] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6128.999020] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6130.719281] Key type lgssc registered [ 6132.703641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6143.060671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.110323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.842705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.779652] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6154.209567] LustreError: 184602:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.223479] LustreError: 184602:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6159.333427] LustreError: 184603:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.482897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.158228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6166.575536] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6176.262271] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.012692] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.504474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.000646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.413253] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6210.350328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.525605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6235.111649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.111842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.125850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6235.134375] Lustre: Skipped 1 previous similar message [ 6240.231181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6240.248323] Lustre: Skipped 2 previous similar messages [ 6240.645508] Lustre: server umount lustre-OST0000 complete [ 6248.416603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.422043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.438845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6250.466922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6250.467365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6251.918757] Lustre: server umount lustre-MDT0000 complete [ 6255.587770] LustreError: 187104:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.609619] LustreError: 187104:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6256.202842] LustreError: 184584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853230 with bad export cookie 12169310882252984201 [ 6256.211325] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.220702] LustreError: 184584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6257.014651] Lustre: server umount lustre-MDT0001 complete [ 6270.074796] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6273.850207] Key type lgssc unregistered [ 6274.211657] LNet: 188165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6274.221655] LNetError: 188165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6275.318710] LNet: Removed LNI 192.168.204.146@tcp [ 6276.548438] Key type .llcrypt unregistered [ 6276.551232] Key type ._llcrypt unregistered [ 6309.011662] Key type ._llcrypt registered [ 6309.014960] Key type .llcrypt registered [ 6309.359044] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6310.947497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6311.018266] alg: No test for adler32 (adler32-zlib) [ 6312.464977] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6313.100797] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6314.895201] Key type lgssc registered [ 6316.615478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6327.989651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.014062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.521324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.439739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6339.063800] LustreError: 189586:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.088519] LustreError: 189586:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6344.182320] LustreError: 189587:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.098910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.527289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.115833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6361.284414] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6369.227461] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.233462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.723320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.230353] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6395.129502] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6399.232653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6406.629456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.639816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.658042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.294982] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 6409.703972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.704305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.722417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.372906] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 6412.379906] Lustre: Skipped 1 previous similar message [ 6412.712560] Lustre: server umount lustre-OST0000 complete [ 6421.676272] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f3606255000.ost_server_uuid 50 [ 6423.631871] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6438.324905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.363466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.689360] Lustre: server umount lustre-MDT0000 complete [ 6449.344842] LustreError: 189568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853423 with bad export cookie 12094928672758018318 [ 6449.355233] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.363421] LustreError: 189568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6449.898989] Lustre: server umount lustre-MDT0001 complete [ 6460.149147] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:57:12 (1787853432) [ 6467.170454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.763880] LustreError: 193504:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.851629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.188821] LustreError: 193505:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.288455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6477.283830] LustreError: 193504:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.261352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.760049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.604968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6497.650761] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.475083] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.520902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.705262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.754694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6519.047560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6531.894225] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.792369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.078513] Lustre: lustre-OST0001: new disk, initializing [ 6543.080435] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6543.085669] Lustre: 196350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6543.181970] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6548.644825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6548.658377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6548.792921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6550.321702] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6560.519853] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.325890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.334693] Lustre: Skipped 1 previous similar message [ 6564.339449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6564.342568] Lustre: Skipped 5 previous similar messages [ 6564.347728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.652333] Lustre: server umount lustre-OST0000 complete [ 6571.584147] LustreError: 195328:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.623431] LustreError: 195328:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6580.659486] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6580.967342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6589.096997] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6589.751036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6589.938667] LustreError: 196019:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.016843] LustreError: 196019:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 6610.401286] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6610.420812] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.429866] LustreError: Skipped 1 previous similar message [ 6610.433511] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6610.454739] Lustre: Skipped 2 previous similar messages [ 6610.477900] Lustre: Skipped 5 previous similar messages [ 6612.389168] Lustre: server umount lustre-OST0001 complete [ 6617.056598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6620.641175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.653544] Lustre: Skipped 1 previous similar message [ 6623.030378] Lustre: server umount lustre-MDT0000 complete [ 6625.773296] LustreError: 197810:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.803890] LustreError: 197810:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 6627.302933] LustreError: 193484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853601 with bad export cookie 12094928672758019389 [ 6627.324530] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.330563] LustreError: 193484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6627.680068] Lustre: server umount lustre-MDT0001 complete [ 6656.472154] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:00:28 (1787853628) [ 6664.229842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.594624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6664.683678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6664.917628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.072109] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6682.284189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.443630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6682.536731] Lustre: 199793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6688.938392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6698.025232] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6705.677274] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6713.696730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.976466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6721.219487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6722.590728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6732.817806] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6740.324224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.412447] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6740.684864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.697411] Lustre: Skipped 2 previous similar messages [ 6748.217648] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6759.128388] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6763.504903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6763.534740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6763.548040] Lustre: Skipped 4 previous similar messages [ 6768.729493] Lustre: server umount lustre-OST0000 complete [ 6770.256955] LustreError: 201656:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.089329] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6789.111075] LustreError: Skipped 1 previous similar message [ 6791.945310] Lustre: server umount lustre-OST0001 complete [ 6799.333103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.349273] Lustre: Skipped 4 previous similar messages [ 6802.615994] Lustre: server umount lustre-MDT0000 complete [ 6806.458824] LustreError: 200550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853780 with bad export cookie 12094928672758020796 [ 6806.469879] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6806.477062] LustreError: 200550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6806.874849] Lustre: server umount lustre-MDT0001 complete [ 6835.562910] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:03:27 (1787853807) [ 6855.168854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6865.950470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6911.228430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.544702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6911.568486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6911.613197] Lustre: lustre-MDT0000: new disk, initializing [ 6911.672928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.686164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6916.823852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6928.714129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.887388] Lustre: 207906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6929.021878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6929.036080] Lustre: Skipped 1 previous similar message [ 6929.173624] Lustre: lustre-MDT0001: new disk, initializing [ 6929.382638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6929.406794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6934.853745] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6945.712266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.495992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6960.512359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.789643] Lustre: lustre-OST0000: new disk, initializing [ 6960.792855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6960.798288] Lustre: 209872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6967.457805] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6969.380380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6969.392174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6969.483066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6977.401611] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.208737] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6982.407922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6987.056836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6987.288216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6989.796337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6989.806262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.827087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.833090] Lustre: Skipped 7 previous similar messages [ 6995.707070] Lustre: server umount lustre-OST0000 complete [ 7006.702106] Lustre: server umount lustre-MDT0000 complete [ 7010.789310] LustreError: 211210:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.837848] LustreError: 211210:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 13 previous similar messages [ 7011.040794] LustreError: 207897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853985 with bad export cookie 12094928672758022357 [ 7011.061360] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7011.476903] Lustre: server umount lustre-MDT0001 complete [ 7020.713417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.718986] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7034.147207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.909164] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7046.502469] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7053.047947] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7059.452509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.879076] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7073.509273] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7078.389935] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7078.567296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7083.148171] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7083.350790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7087.588576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.597790] LustreError: Skipped 1 previous similar message [ 7091.736542] Lustre: server umount lustre-OST0000 complete [ 7098.236609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7103.146777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7132.851894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.497795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7139.516391] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7140.143770] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7147.680208] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.607930] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7155.171328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7155.175862] LustreError: Skipped 1 previous similar message [ 7155.184838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.195778] Lustre: Skipped 5 previous similar messages [ 7155.203368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7155.209469] Lustre: Skipped 11 previous similar messages [ 7160.526363] Lustre: server umount lustre-OST0000 complete [ 7171.334181] Lustre: server umount lustre-MDT0000 complete [ 7175.531714] LustreError: 212070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854149 with bad export cookie 12094928672758023239 [ 7175.538417] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7175.542821] LustreError: 212070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7175.875598] Lustre: server umount lustre-MDT0001 complete [ 7183.924023] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:09:16 (1787854156) [ 7191.035280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.490980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.496712] Lustre: Skipped 6 previous similar messages [ 7196.192398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7205.216526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.619919] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7219.066927] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7225.310428] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7233.308953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.254344] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7248.015190] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7253.083378] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7253.350595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7258.376949] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7258.565558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7264.931435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.094578] Lustre: lustre-OST0001: new disk, initializing [ 7265.097868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7265.102835] Lustre: 221427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7271.758399] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7274.066981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7274.079480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7274.192058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7282.414816] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7287.492462] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7287.686584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7292.422501] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7292.666374] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7294.945519] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7309.279214] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7309.352643] Lustre: server umount lustre-OST0001 complete [ 7310.321892] LustreError: 219879:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.344026] LustreError: 219879:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 7315.751196] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7315.965688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7320.893337] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7321.185970] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7350.442892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.890569] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7355.904597] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7355.910572] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7355.942791] Lustre: Skipped 1 previous similar message [ 7356.419465] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7364.594206] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.402352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7374.813687] Lustre: server umount lustre-OST0001 complete [ 7398.425127] Lustre: server umount lustre-OST0000 complete [ 7412.193250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.199093] Lustre: Skipped 21 previous similar messages [ 7416.288962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7416.570097] Lustre: server umount lustre-MDT0000 complete [ 7420.309958] LustreError: 218036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854394 with bad export cookie 12094928672758024198 [ 7420.315211] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.326074] LustreError: 218036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7426.655758] Lustre: server umount lustre-MDT0001 complete [ 7454.232460] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:13:46 (1787854426) [ 7460.668908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.083086] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7461.126751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7466.303710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7475.846936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.010199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7476.049197] Lustre: 226673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7481.362760] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7489.715988] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.598743] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7503.988435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.172756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7510.582563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7518.949406] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7528.307306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7528.396184] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7534.888633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7542.654294] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.337789] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7550.608649] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7550.887553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7556.494846] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7556.806960] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7563.923877] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f3602a30000.ost_server_uuid 50 [ 7565.548756] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f3602a30000.ost_server_uuid in IDLE state after 0 sec [ 7566.594036] Lustre: Permanently deactivating lustre-OST0001 [ 7566.607523] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7570.361719] Lustre: Permanently reactivating lustre-OST0001 [ 7578.085897] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.096315] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.106083] LustreError: Skipped 5 previous similar messages [ 7578.135556] Lustre: Skipped 10 previous similar messages [ 7581.374713] Lustre: server umount lustre-OST0001 complete [ 7592.354272] Lustre: server umount lustre-OST0000 complete [ 7602.767800] Lustre: server umount lustre-MDT0000 complete [ 7607.738598] LustreError: 226667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854581 with bad export cookie 12094928672758025479 [ 7607.753395] LustreError: 226667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7607.756828] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7608.553288] Lustre: server umount lustre-MDT0001 complete [ 7634.814761] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:16:47 (1787854607) [ 7643.118770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.662816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7643.703271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7643.718269] Lustre: Skipped 5 previous similar messages [ 7648.898622] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7658.072745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.228954] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7658.260674] Lustre: 233368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7663.260257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7671.570639] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7677.648615] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7685.225396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.481851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7685.505986] Lustre: Permanently deactivating lustre-OST0000 [ 7685.521765] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7692.437390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7692.857609] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7699.492813] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.479165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.532779] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7706.723403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.730316] Lustre: Skipped 11 previous similar messages [ 7712.836761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7714.951688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7720.337190] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7722.355834] Lustre: Permanently reactivating lustre-OST0000 [ 7722.380412] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7722.392646] Lustre: Skipped 2 previous similar messages [ 7729.223247] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7729.223816] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000163c784e, cur 1787854703 deadline 1787854696 last 1787854666 [ 7729.233124] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7729.233574] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7729.310745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7741.723566] Lustre: server umount lustre-OST0001 complete [ 7752.827141] Lustre: server umount lustre-OST0000 complete [ 7764.784863] Lustre: server umount lustre-MDT0000 complete [ 7768.276768] LustreError: 234108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854742 with bad export cookie 12094928672758027131 [ 7768.279566] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.288557] LustreError: 234108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7768.716845] Lustre: server umount lustre-MDT0001 complete [ 7780.905509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7783.855450] Key type lgssc unregistered [ 7784.126977] LNet: 238255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.138187] LNetError: 238255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7784.150853] LNet: Removed LNI 192.168.204.146@tcp [ 7785.094665] Key type .llcrypt unregistered [ 7785.098853] Key type ._llcrypt unregistered [ 7808.755354] Key type ._llcrypt registered [ 7808.757522] Key type .llcrypt registered [ 7808.869446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7810.127714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.157250] alg: No test for adler32 (adler32-zlib) [ 7811.158822] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7811.346977] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7813.008913] Key type lgssc registered [ 7814.245219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7823.470585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7823.504561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.960500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.269222] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7834.094765] LustreError: 239722:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7834.117334] LustreError: 239722:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7839.201256] LustreError: 239721:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.000798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.131924] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7840.163204] Lustre: 239710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7840.176576] Lustre: Permanently deactivating lustre-MDT0001 [ 7840.190545] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7840.537240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7841.428874] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7841.435075] LustreError: 239752:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7841.447730] LustreError: 239752:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7841.461673] LustreError: 239752:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7845.320699] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7852.735382] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7858.936466] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7866.874170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7867.305502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.355320] LustreError: 241561:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.378542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7874.544265] LustreError: 241563:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.783404] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7879.651661] LustreError: 241562:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.744969] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7890.609114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7890.792061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.051908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7896.059556] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7896.738432] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7904.865989] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7908.939332] Lustre: Permanently reactivating lustre-MDT0001 [ 7908.965378] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7908.971887] Lustre: Skipped 2 previous similar messages [ 7910.597287] Lustre: lustre-MDT0001: Client 54220634-ca1c-43ff-bc72-e56d96ecb690 (at 192.168.204.46@tcp) reconnecting [ 7918.499667] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7920.404353] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7926.753432] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7927.022874] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7932.484166] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7932.800761] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7935.483159] Lustre: Permanently deactivating lustre-MDT0001 [ 7935.509550] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7935.520834] Lustre: Skipped 2 previous similar messages [ 7941.748609] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7951.332398] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.339398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7951.346560] Lustre: Skipped 3 previous similar messages [ 7951.354516] Lustre: Skipped 3 previous similar messages [ 7953.045758] Lustre: server umount lustre-MDT0000 complete [ 7956.451203] LustreError: 239718:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7956.480597] LustreError: 239718:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 7956.645273] LustreError: 239702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854930 with bad export cookie 3346837420255051642 [ 7956.657231] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.657738] LustreError: 239702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7956.972791] Lustre: server umount lustre-MDT0001 complete [ 7962.738371] Lustre: server umount lustre-OST0000 complete [ 7974.131658] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:22:26 (1787854946) [ 7985.855152] Lustre: server umount lustre-OST0001 complete [ 7992.827301] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8004.110739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8043.589784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.923200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8043.935593] Lustre: Skipped 2 previous similar messages [ 8043.970905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8044.087961] Lustre: lustre-MDT0000: new disk, initializing [ 8044.223663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.246584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8048.736359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8059.781928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.955466] Lustre: 247978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8060.001927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8060.005307] Lustre: Skipped 1 previous similar message [ 8060.060785] Lustre: lustre-MDT0001: new disk, initializing [ 8060.146374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8060.181805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8060.193529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8064.916147] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8075.939923] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8082.628485] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8090.440685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8090.651769] Lustre: lustre-OST0000: new disk, initializing [ 8090.658823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8090.664829] Lustre: 249947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8090.742657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8096.698972] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8098.408830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8098.418646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8098.598836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8107.112385] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8109.737425] LustreError: 247986:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8119.080268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.354269] Lustre: lustre-OST0001: new disk, initializing [ 8119.360427] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8119.379604] Lustre: 251060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8119.807116] LustreError: 247986:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8119.834547] LustreError: 249464:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8126.799754] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8127.545821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8127.568457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8127.655044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8129.927361] LustreError: 249464:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8136.895732] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.764622] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8142.825144] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8142.827287] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8142.835978] Lustre: Skipped 1 previous similar message [ 8142.845069] Lustre: Skipped 1 previous similar message [ 8147.091565] Lustre: server umount lustre-OST0001 complete [ 8147.958992] LustreError: 249930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.984212] LustreError: 249930:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8165.357873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8165.359511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.369550] LustreError: Skipped 1 previous similar message [ 8165.374493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8165.391276] Lustre: Skipped 1 previous similar message [ 8168.422159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8168.429223] Lustre: Skipped 2 previous similar messages [ 8170.368700] Lustre: server umount lustre-OST0000 complete [ 8178.144829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8178.159546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.188524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.386365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8188.861353] Lustre: server umount lustre-MDT0000 complete [ 8188.898973] LustreError: 247990:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.933380] LustreError: 247990:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 8193.044832] LustreError: 247970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855167 with bad export cookie 3346837420255054239 [ 8193.051979] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.058665] LustreError: 247970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8199.480255] Lustre: server umount lustre-MDT0001 complete [ 8212.455551] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8215.462447] Key type lgssc unregistered [ 8215.802248] LNet: 253097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8215.807465] LNetError: 253097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8215.820814] LNet: Removed LNI 192.168.204.146@tcp [ 8216.736227] Key type .llcrypt unregistered [ 8216.738142] Key type ._llcrypt unregistered [ 8260.003466] Key type ._llcrypt registered [ 8260.005551] Key type .llcrypt registered [ 8260.198857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8261.569639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8261.646532] alg: No test for adler32 (adler32-zlib) [ 8262.786720] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8263.052886] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8264.743243] Key type lgssc registered [ 8265.761683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8273.515606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.541613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.799271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8274.812662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8274.967753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8279.659946] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8289.554344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.669488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8289.750375] Lustre: 255307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8290.023945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8295.292220] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8302.952369] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8308.589908] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8314.459439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.676288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8314.924560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.211463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8321.453836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8328.525527] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8382.435495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8382.442920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.454311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8382.950629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.971907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8382.973761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8385.325272] Lustre: server umount lustre-OST0000 complete [ 8389.788946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8403.657925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.160407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.340607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8421.575342] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8423.546846] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8423.557316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8423.557883] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8464.879086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.890433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8464.901668] Lustre: Skipped 1 previous similar message [ 8464.902650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.925823] Lustre: Skipped 1 previous similar message [ 8468.970349] Lustre: server umount lustre-OST0000 complete [ 8474.595833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8474.603658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.631504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.225643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.239265] Lustre: Skipped 3 previous similar messages [ 8487.391274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8487.522433] Lustre: server umount lustre-MDT0000 complete [ 8491.073228] LustreError: 255301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855465 with bad export cookie 14172117512450514492 [ 8491.074868] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8491.087364] LustreError: 255301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8491.443936] Lustre: server umount lustre-MDT0001 complete [ 8503.940099] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8507.158381] Key type lgssc unregistered [ 8507.470236] LNet: 260744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8507.481648] LNetError: 260744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8507.494357] LNet: Removed LNI 192.168.204.146@tcp [ 8508.416934] Key type .llcrypt unregistered [ 8508.423162] Key type ._llcrypt unregistered [ 8534.416948] Key type ._llcrypt registered [ 8534.426581] Key type .llcrypt registered [ 8534.707958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8536.298253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8536.517377] alg: No test for adler32 (adler32-zlib) [ 8537.599491] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8537.804813] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8539.443304] Key type lgssc registered [ 8540.522671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8550.366505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8550.385583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.027768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.644563] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8561.122521] LustreError: 262170:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8561.154059] LustreError: 262170:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8566.253062] LustreError: 262169:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.170821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.464566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8575.368437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8584.885975] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8593.799500] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8602.070700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8602.340147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.305809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8619.772525] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8622.851729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8653.792993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.806567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.813653] Lustre: Skipped 1 previous similar message [ 8653.832558] Lustre: Skipped 1 previous similar message [ 8658.923206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8658.942089] Lustre: Skipped 1 previous similar message [ 8660.069150] Lustre: server umount lustre-OST0000 complete [ 8665.574157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8665.599310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.629571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.155052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.164573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.978463] Lustre: server umount lustre-MDT0000 complete [ 8674.280728] LustreError: 262170:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8674.318268] LustreError: 262170:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8677.003245] LustreError: 262150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855651 with bad export cookie 11460870859445060998 [ 8677.016165] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.018831] LustreError: 262150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8677.535494] Lustre: server umount lustre-MDT0001 complete [ 8689.970251] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8693.549584] Key type lgssc unregistered [ 8693.881753] LNet: 266482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.896862] LNetError: 266482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8693.912653] LNet: Removed LNI 192.168.204.146@tcp [ 8694.955686] Key type .llcrypt unregistered [ 8694.961908] Key type ._llcrypt unregistered [ 8713.065804] Key type ._llcrypt registered [ 8713.071378] Key type .llcrypt registered [ 8713.235520] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8714.430057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8714.493534] alg: No test for adler32 (adler32-zlib) [ 8715.765379] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8716.110505] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8717.826775] Key type lgssc registered [ 8719.298328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8728.978781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8729.008400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.694355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.263469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8740.832280] LustreError: 267704:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.843420] LustreError: 267704:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8745.953342] LustreError: 267705:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.355764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.809156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8752.983083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8761.779840] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8768.419780] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8777.124989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8777.525406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.120419] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8794.652681] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8800.059442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8813.537988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8813.554956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.585185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8815.595102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8815.596892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8815.610413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.707088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8820.723258] Lustre: Skipped 2 previous similar messages [ 8825.846337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8825.851151] Lustre: Skipped 1 previous similar message [ 8827.359169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8827.566263] Lustre: server umount lustre-OST0000 complete [ 8835.043674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8835.055335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.063133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.718488] Lustre: server umount lustre-MDT0000 complete [ 8841.191725] LustreError: 267699:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.204951] LustreError: 267699:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8842.150234] LustreError: 268442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855816 with bad export cookie 8499494833893465656 [ 8842.154572] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8842.169698] LustreError: 268442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8842.550626] Lustre: server umount lustre-MDT0001 complete [ 8854.270383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8857.876222] Key type lgssc unregistered [ 8858.166440] LNet: 271395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8858.183221] LNetError: 271395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8858.210869] LNet: Removed LNI 192.168.204.146@tcp [ 8859.148613] Key type .llcrypt unregistered [ 8859.152440] Key type ._llcrypt unregistered [ 8884.891804] Key type ._llcrypt registered [ 8884.894795] Key type .llcrypt registered [ 8885.057345] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8886.676014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8886.740340] alg: No test for adler32 (adler32-zlib) [ 8887.986215] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8888.356866] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8890.073057] Key type lgssc registered [ 8891.575702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8901.303823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8901.343225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.080134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8908.956855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8913.390158] LustreError: 272816:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.407497] LustreError: 272816:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8917.251713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.845551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8922.540749] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8931.567894] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8939.626765] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8949.229509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.923526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8960.675740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8972.196266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8973.573601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9003.495342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9003.521616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.546535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9004.522021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9004.529037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.535402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9004.554419] Lustre: Skipped 1 previous similar message [ 9006.176622] Lustre: server umount lustre-OST0000 complete [ 9010.159767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9010.168395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.196295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9014.755014] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.757191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9014.792447] Lustre: Skipped 1 previous similar message [ 9016.699565] Lustre: server umount lustre-MDT0000 complete [ 9019.873103] LustreError: 273544:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.901864] LustreError: 273544:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9021.448502] LustreError: 272798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855995 with bad export cookie 2618474705271460428 [ 9021.458492] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.468903] LustreError: 272798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9021.877648] Lustre: server umount lustre-MDT0001 complete [ 9034.472694] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9037.773467] Key type lgssc unregistered [ 9038.099492] LNet: 277129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9038.113961] LNetError: 277129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9038.149452] LNet: Removed LNI 192.168.204.146@tcp [ 9039.508821] Key type .llcrypt unregistered [ 9039.513249] Key type ._llcrypt unregistered [ 9058.333151] Key type ._llcrypt registered [ 9058.335884] Key type .llcrypt registered [ 9058.576370] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9059.722621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9059.788865] alg: No test for adler32 (adler32-zlib) [ 9060.914464] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 9061.205154] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9062.951508] Key type lgssc registered [ 9064.423297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9074.326987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9074.351838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.180478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.929637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9086.444143] LustreError: 278368:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9086.461181] LustreError: 278368:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9091.564637] LustreError: 278370:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.725479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.158516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9098.473509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9106.698212] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9114.147939] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9121.189088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.750639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.835647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9129.200738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9138.041711] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9153.504599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9153.522710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.530429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.547607] Lustre: Skipped 1 previous similar message [ 9153.557087] Lustre: Skipped 1 previous similar message [ 9156.328481] Lustre: server umount lustre-OST0000 complete [ 9163.756983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9163.774280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9163.782487] Lustre: Skipped 1 previous similar message [ 9167.379087] Lustre: server umount lustre-MDT0000 complete [ 9168.867911] LustreError: 278361:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9168.899179] LustreError: 278361:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9171.742704] LustreError: 278333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856145 with bad export cookie 4023224260253394709 [ 9171.744495] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9171.759881] LustreError: 278333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9172.068273] Lustre: server umount lustre-MDT0001 complete [ 9187.102541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9191.241303] Key type lgssc unregistered [ 9191.608644] LNet: 282076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9191.614712] LNetError: 282076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9191.656525] LNet: Removed LNI 192.168.204.146@tcp [ 9192.703685] Key type .llcrypt unregistered [ 9192.707443] Key type ._llcrypt unregistered [ 9205.444320] Key type ._llcrypt registered [ 9205.449603] Key type .llcrypt registered [ 9205.623143] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9226.067243] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9242.880588] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9243.902119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9244.071702] alg: No test for adler32 (adler32-zlib) [ 9245.156667] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 9245.405056] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9247.071446] Key type lgssc registered [ 9248.310783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9288.645393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9288.715829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9290.048817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9290.087583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9290.220490] Lustre: lustre-MDT0000: new disk, initializing [ 9290.365982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.395873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9296.174089] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9308.216217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.339278] Lustre: 285632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9308.392715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9308.397287] Lustre: Skipped 1 previous similar message [ 9308.490588] Lustre: lustre-MDT0001: new disk, initializing [ 9308.651563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9308.706117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9308.727028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9314.328848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9327.128288] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9334.329327] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9342.218284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.498406] Lustre: lustre-OST0000: new disk, initializing [ 9342.503941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9342.515983] Lustre: 287599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9342.634980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.688249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9348.709376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9348.828635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9350.917136] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9360.983759] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9366.136841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9366.309411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9371.504277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9371.721310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9374.691105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9374.706113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.708796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9374.726431] LustreError: Skipped 1 previous similar message [ 9379.834523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.849062] Lustre: Skipped 2 previous similar messages [ 9380.172678] Lustre: server umount lustre-OST0000 complete [ 9384.929424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9384.937969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.950186] Lustre: Skipped 2 previous similar messages [ 9390.051906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.068418] Lustre: Skipped 2 previous similar messages [ 9390.839103] Lustre: server umount lustre-MDT0000 complete [ 9395.372056] LustreError: 285623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856369 with bad export cookie 6752110276030655643 [ 9395.376460] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.396344] LustreError: 285623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9395.885885] Lustre: server umount lustre-MDT0001 complete [ 9406.286075] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:46:18 (1787856378) [ 9415.766540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.183333] LustreError: 290016:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.213627] LustreError: 290016:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9416.322156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.795292] LustreError: 290015:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.359632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9426.923616] LustreError: 290016:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.032284] LustreError: 290015:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.518496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.261190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9440.975482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9451.105470] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9456.611752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.186903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9464.719421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.594673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9479.541800] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9495.525234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9495.529924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9495.540558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.547579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9501.138487] Lustre: server umount lustre-OST0000 complete [ 9506.795326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.827284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9506.835795] Lustre: Skipped 2 previous similar messages [ 9506.863609] Lustre: Skipped 3 previous similar messages [ 9509.183671] Lustre: server umount lustre-MDT0000 complete [ 9511.904263] LustreError: 290016:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.932577] LustreError: 290016:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9513.852465] LustreError: 289997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856487 with bad export cookie 6752110276030656525 [ 9513.861824] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.870128] LustreError: 289997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9514.154709] Lustre: server umount lustre-MDT0001 complete [ 9525.856387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9528.511256] Key type lgssc unregistered [ 9528.780813] LNet: 293578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9528.786334] LNetError: 293578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9528.808538] LNet: Removed LNI 192.168.204.146@tcp [ 9529.648261] Key type .llcrypt unregistered [ 9529.650106] Key type ._llcrypt unregistered [ 9560.342200] Key type ._llcrypt registered [ 9560.343661] Key type .llcrypt registered [ 9560.442850] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9577.326319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9578.248696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9578.307492] alg: No test for adler32 (adler32-zlib) [ 9579.335988] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 9579.536984] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9581.271253] Key type lgssc registered [ 9582.425670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9629.914853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9629.937928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9631.176900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9631.202809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9631.293126] Lustre: lustre-MDT0000: new disk, initializing [ 9631.420655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9631.436640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9635.066389] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9645.067976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.188299] Lustre: 297407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9645.245918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9645.248800] Lustre: Skipped 1 previous similar message [ 9645.307441] Lustre: lustre-MDT0001: new disk, initializing [ 9645.387161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9645.424332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9645.446352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9649.847445] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9660.387102] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9666.119188] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.858136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9674.153904] Lustre: lustre-OST2710: new disk, initializing [ 9674.156834] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9674.164350] Lustre: 299375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9674.251460] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9679.946141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9679.962716] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9680.144865] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9682.680171] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9694.483707] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9701.841837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9701.972267] Lustre: lustre-OST03e8: new disk, initializing [ 9701.974983] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9701.980288] Lustre: 300437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9702.080989] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9708.467308] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9710.750632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9710.762419] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9710.861180] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9716.760779] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9724.099433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9724.373855] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9729.735958] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9730.077031] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9736.092659] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9736.261653] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9741.934323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9742.126101] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9747.936772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9747.952475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.965596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9751.503327] Lustre: server umount lustre-MDT0000 complete [ 9752.052761] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.085085] Lustre: Skipped 1 previous similar message [ 9755.173381] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856729 with bad export cookie 13403657222505201902 [ 9755.177366] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9755.183366] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9755.533513] Lustre: server umount lustre-MDT0001 complete [ 9769.309899] Lustre: server umount lustre-OST2710 complete [ 9783.736467] Lustre: server umount lustre-OST03e8 complete [ 9804.279923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9815.758569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9854.453069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.734146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9854.789128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9854.904961] Lustre: lustre-MDT0000: new disk, initializing [ 9855.039402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.051840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9860.122865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9872.242475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.407493] Lustre: 305693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9872.470096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9872.475782] Lustre: Skipped 1 previous similar message [ 9872.581203] Lustre: lustre-MDT0001: new disk, initializing [ 9872.703535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9872.767816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9872.789058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9878.148919] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9889.674442] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9896.866696] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9905.000636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.443897] Lustre: lustre-OST0000: new disk, initializing [ 9905.459973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9905.477149] Lustre: 307661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9905.582325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.409931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9911.430720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9911.597597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9912.336986] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9922.443224] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9927.480720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9927.732876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9931.876673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9932.075359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9937.386643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.389390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.396253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9937.424498] Lustre: Skipped 2 previous similar messages [ 9939.799419] Lustre: server umount lustre-OST0000 complete [ 9944.552147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9944.564073] LustreError: Skipped 1 previous similar message [ 9944.568214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.583242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.591288] Lustre: Skipped 1 previous similar message [ 9947.631845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.647299] Lustre: Skipped 1 previous similar message [ 9950.808473] Lustre: server umount lustre-MDT0000 complete [ 9952.738169] LustreError: 306828:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.765400] LustreError: 306828:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 9955.245410] LustreError: 305686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856929 with bad export cookie 13403657222505203589 [ 9955.263443] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9955.735971] Lustre: server umount lustre-MDT0001 complete [ 9966.077616] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:55:38 (1787856938) [ 9968.424763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9970.449755] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:55:42 (1787856942) [ 9974.573235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10005.787954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.265733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10006.302486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10006.610458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10011.709950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10020.995963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.134605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10021.178872] Lustre: 311520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10026.819766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10049.003847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.377762] Lustre: Denying initial registration attempt from nid 192.168.204.146@tcp, specified as failover [10049.396836] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10049.416944] LustreError: 312973:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10049.424985] LustreError: 312973:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10049.436621] LustreError: 312973:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10049.544424] Lustre: server umount lustre-OST0000 complete [10049.546298] LustreError: 312973:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10057.184496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10057.194493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.212917] Lustre: Skipped 1 previous similar message [10057.219907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.557556] Lustre: server umount lustre-MDT0000 complete [10064.381916] LustreError: 311531:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.400987] LustreError: 311531:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10068.152782] LustreError: 311510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857042 with bad export cookie 13403657222505204471 [10068.159182] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.164982] LustreError: 311510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10068.550957] Lustre: server umount lustre-MDT0001 complete [10081.110987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10084.881535] Key type lgssc unregistered [10085.197834] LNet: 314095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10085.205345] LNetError: 314095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10085.228119] LNet: Removed LNI 192.168.204.146@tcp [10086.329558] Key type .llcrypt unregistered [10086.333305] Key type ._llcrypt unregistered [10099.630369] Key type ._llcrypt registered [10099.635047] Key type .llcrypt registered [10099.810864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10100.731249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10100.834064] alg: No test for adler32 (adler32-zlib) [10102.016245] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10102.331924] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10104.055416] Key type lgssc registered [10105.146806] Lustre: Echo OBD driver; http://www.lustre.org/ [10154.785186] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10166.620813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10166.657907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.013784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10168.025709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10168.202191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10173.569443] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10181.947902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.081087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10182.141494] Lustre: 316868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10182.405441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10187.531906] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10194.624990] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10201.693263] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10222.595897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10222.886290] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10223.125123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10229.138575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10237.075627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10240.487180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10240.498492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10240.510086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10245.632230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10245.651077] Lustre: Skipped 2 previous similar messages [10246.730765] Lustre: server umount lustre-OST0000 complete [10250.721073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.742589] Lustre: Skipped 1 previous similar message [10250.745426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.843702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.858033] Lustre: Skipped 2 previous similar messages [10256.988942] Lustre: server umount lustre-MDT0000 complete [10261.376460] LustreError: 316861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857235 with bad export cookie 10911185766607556647 [10261.381866] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10261.388710] LustreError: 316861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10261.750201] Lustre: server umount lustre-MDT0001 complete [10273.524125] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10276.644276] Key type lgssc unregistered [10276.963723] LNet: 320447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10276.985540] LNetError: 320447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10277.003254] LNet: Removed LNI 192.168.204.146@tcp [10277.971310] Key type .llcrypt unregistered [10277.977469] Key type ._llcrypt unregistered [10301.485550] Key type ._llcrypt registered [10301.489337] Key type .llcrypt registered [10301.583798] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10302.388136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10302.466992] alg: No test for adler32 (adler32-zlib) [10303.562621] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10303.891668] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10305.575923] Key type lgssc registered [10306.636814] Lustre: Echo OBD driver; http://www.lustre.org/ [10315.103751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10315.117919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.579852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10321.549884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10327.007899] LustreError: 321873:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.022440] LustreError: 321873:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10331.210129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.552647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10337.293812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10345.266610] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10351.930039] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10360.641453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.223549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10368.457848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10377.688918] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10386.915063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.927183] Lustre: Skipped 1 previous similar message [10386.931070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10389.258596] Lustre: server umount lustre-MDT0000 complete [10390.126670] LustreError: 323555:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.035476] LustreError: 321874:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.064849] LustreError: 321874:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10393.496831] LustreError: 321853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857367 with bad export cookie 5568642767210361124 [10393.503100] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.506988] LustreError: 321853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10395.205799] LustreError: 321870:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.235857] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [10395.258550] Lustre: Skipped 2 previous similar messages [10397.156093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.177451] Lustre: Skipped 1 previous similar message [10397.188616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10400.080896] Lustre: server umount lustre-MDT0001 complete [10407.137079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10417.014611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.207738] LustreError: 325451:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10418.221887] LustreError: 325451:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000003f1d82c5 x1874704232119808/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787857392 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10418.253781] LustreError: 325451:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10418.659448] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4d47cb17c49f4d08 [10418.667645] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [10418.924309] LustreError: 323705:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.939577] LustreError: 323705:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10419.024674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10419.078918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10423.622076] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10428.268706] LustreError: 325465:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.271431] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10428.284094] LustreError: 325465:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10432.165579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.550801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10436.852595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10437.648818] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10444.578959] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10446.309580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10453.215386] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10454.777930] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10463.201415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10463.206785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10463.211767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10463.219168] Lustre: Skipped 1 previous similar message [10463.224877] Lustre: Skipped 2 previous similar messages [10466.932078] Lustre: server umount lustre-OST0000 complete [10473.442085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10473.444177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10473.458546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10473.473700] Lustre: Skipped 1 previous similar message [10473.494880] Lustre: Skipped 1 previous similar message [10477.321983] Lustre: server umount lustre-MDT0000 complete [10478.564905] LustreError: 325469:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.592946] LustreError: 325469:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10480.646477] LustreError: 324656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857454 with bad export cookie 5568642767210368264 [10480.648277] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10480.656349] LustreError: 324656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10480.936607] Lustre: server umount lustre-MDT0001 complete [10491.978263] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10495.288655] Key type lgssc unregistered [10495.602465] LNet: 328146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10495.612991] LNetError: 328146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10495.653332] LNet: Removed LNI 192.168.204.146@tcp [10496.642425] Key type .llcrypt unregistered [10496.647170] Key type ._llcrypt unregistered [10524.271445] Key type ._llcrypt registered [10524.273700] Key type .llcrypt registered [10524.376327] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10525.903757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10525.943994] alg: No test for adler32 (adler32-zlib) [10527.108934] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10527.385311] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10529.031214] Key type lgssc registered [10530.691573] Lustre: Echo OBD driver; http://www.lustre.org/ [10540.690479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10540.700222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10542.317506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10546.852663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10552.808357] LustreError: 329565:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.829963] LustreError: 329565:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10556.056553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10556.593758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10561.276044] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10570.292931] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10577.208489] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10581.932073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.949743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.959828] Lustre: Skipped 1 previous similar message [10587.107318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.110290] Lustre: Skipped 1 previous similar message [10592.226260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10592.230951] Lustre: Skipped 1 previous similar message [10595.295123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10595.428383] Lustre: server umount lustre-MDT0000 complete [10597.347213] LustreError: 329565:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10597.371601] LustreError: 329565:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10599.627962] LustreError: 329544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857573 with bad export cookie 1183415746149869620 [10599.636230] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.644693] LustreError: 329544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10599.938352] Lustre: server umount lustre-MDT0001 complete [10616.559977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.739755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10616.761516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10616.967565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10621.733765] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10630.470965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.612182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10630.685829] Lustre: 332154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10630.912422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10635.982316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10644.760122] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10650.613449] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10661.227206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.472138] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10661.487991] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10661.492923] LustreError: 334038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10661.497527] LustreError: 334038:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10661.501268] LustreError: 334038:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10661.568245] Lustre: server umount lustre-OST0000 complete [10661.572625] LustreError: 334038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10672.640341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10673.039028] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10673.168219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10679.013760] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10679.997977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10687.009639] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10695.308693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10695.403561] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10695.473199] Lustre: lustre-OST0001: new disk, initializing [10695.475733] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10695.482581] Lustre: 335580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10695.598368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10701.971596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10701.987941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10702.137877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10703.177640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10711.170609] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10717.678958] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.683242] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10717.690826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10717.691565] Lustre: Skipped 2 previous similar messages [10717.695865] LustreError: Skipped 1 previous similar message [10720.203717] Lustre: server umount lustre-OST0001 complete [10722.803653] LustreError: 334507:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.827319] LustreError: 334507:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10725.863462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10725.882244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.896649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10725.898820] Lustre: Skipped 1 previous similar message [10727.923793] LustreError: 334507:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10727.954566] LustreError: 334507:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10730.817694] Lustre: server umount lustre-OST0000 complete [10738.150128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10738.183222] Lustre: Skipped 1 previous similar message [10738.191624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10738.202630] Lustre: Skipped 4 previous similar messages [10741.790527] Lustre: server umount lustre-MDT0000 complete [10743.264860] LustreError: 333415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.280148] LustreError: 333415:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10746.379804] LustreError: 332145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857720 with bad export cookie 1183415746149870173 [10746.381474] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10746.390737] LustreError: 332145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10746.755245] Lustre: server umount lustre-MDT0001 complete [10760.324498] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10764.325584] Key type lgssc unregistered [10764.655467] LNet: 337542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10764.682196] LNetError: 337542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10764.701273] LNet: Removed LNI 192.168.204.146@tcp [10766.029059] Key type .llcrypt unregistered [10766.033704] Key type ._llcrypt unregistered [10843.633557] Key type ._llcrypt registered [10843.636323] Key type .llcrypt registered [10843.738387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10858.708910] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10859.637818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10859.678802] alg: No test for adler32 (adler32-zlib) [10860.915976] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10861.363754] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10863.119601] Key type lgssc registered [10864.887604] Lustre: Echo OBD driver; http://www.lustre.org/ [10911.762662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10911.819487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.075176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10913.139258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10913.221761] Lustre: lustre-MDT0000: new disk, initializing [10913.297049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10913.310430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10919.421124] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10934.117187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.283633] Lustre: 342479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10934.384931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10934.400300] Lustre: Skipped 1 previous similar message [10934.554675] Lustre: lustre-MDT0001: new disk, initializing [10934.760793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10934.850627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10934.865776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10940.888918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10953.260483] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10959.942614] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10968.197395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10968.612953] Lustre: lustre-OST0000: new disk, initializing [10968.618767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10968.628158] Lustre: 344452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10968.781054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10974.784863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10974.799808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10974.914881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10976.224217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10988.079360] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10994.602351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10994.969655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11000.662173] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11000.935074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11005.921465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11005.922479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11005.930148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11005.936559] LustreError: Skipped 1 previous similar message [11005.981496] Lustre: Skipped 1 previous similar message [11009.646564] Lustre: server umount lustre-OST0000 complete [11016.162898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.167083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11016.183032] Lustre: Skipped 1 previous similar message [11016.209509] Lustre: Skipped 2 previous similar messages [11020.689163] Lustre: server umount lustre-MDT0000 complete [11024.356874] LustreError: 343276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857998 with bad export cookie 10214297402841661279 [11024.363470] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11024.368623] LustreError: 343276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11024.843988] Lustre: server umount lustre-MDT0001 complete [11033.869548] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:13:26 (1787858006) [11043.344807] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:13:35 (1787858015) [11053.980099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.670047] LustreError: 347206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.697885] LustreError: 347206:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11054.767853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.202988] LustreError: 347207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.636978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11064.293471] LustreError: 347206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.413199] LustreError: 347207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.967539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.496682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11078.496223] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11088.088875] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11094.288754] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.079563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11102.792710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11110.459775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11121.477259] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11134.245985] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11136.356291] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11138.130776] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11139.904040] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11143.026468] Lustre: Failing over lustre-MDT0000 [11144.164700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11144.174420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.191362] LustreError: 347936:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.290747] Lustre: server umount lustre-MDT0000 complete [11152.963194] LustreError: 347202:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.979208] LustreError: 347202:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [11160.032881] Lustre: 340702:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858118/real 1787858118] req@000000000946aed7 x1874704817270016/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787858134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11160.074755] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11166.104467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11169.344069] LustreError: 347202:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.367617] LustreError: 347202:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [11170.272331] LustreError: 340701:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000030fbf3e0 x1874704817274624/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11170.850372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11170.918667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11174.462373] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11175.920324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11175.955580] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11176.015487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11177.440269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11190.816293] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11192.833498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11197.184197] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11200.500833] Lustre: Failing over lustre-MDT0000 [11201.173397] Lustre: server umount lustre-MDT0000 complete [11201.521581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11201.529183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11201.538680] LustreError: 347207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.538695] LustreError: 347207:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11201.546467] Lustre: Skipped 3 previous similar messages [11217.902048] Lustre: 340702:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858175/real 1787858175] req@00000000d949ec5d x1874704817292288/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787858191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11217.941601] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11221.173544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.113661] LustreError: 340701:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ba8e064 x1874704817297920/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11227.629433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11227.714267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11228.266294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11232.757314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11232.769700] Lustre: Skipped 2 previous similar messages [11232.821711] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11232.906490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11233.336774] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11248.422541] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11250.422826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11258.338793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11258.360245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11258.371289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11258.386062] Lustre: Skipped 2 previous similar messages [11258.414764] Lustre: Skipped 2 previous similar messages [11263.456318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.066765] Lustre: server umount lustre-MDT0000 complete [11268.602282] LustreError: 347206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11268.645592] LustreError: 347206:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 28 previous similar messages [11269.784075] LustreError: 347186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858243 with bad export cookie 10214297402841663995 [11269.789289] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11269.827711] LustreError: 347186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11270.738518] Lustre: server umount lustre-MDT0001 complete [11287.589578] Lustre: server umount lustre-OST0000 complete [11306.099978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.898836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11312.566240] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11322.425353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.028855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11339.832530] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11347.595986] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11356.747614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.367978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11374.987510] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11376.767162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11387.364219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11387.384776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11387.393274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11387.416247] Lustre: Skipped 2 previous similar messages [11387.421928] Lustre: Skipped 2 previous similar messages [11393.008312] Lustre: server umount lustre-OST0000 complete [11399.660780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11399.676511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11399.690230] Lustre: Skipped 3 previous similar messages [11404.489172] Lustre: server umount lustre-MDT0000 complete [11407.847811] LustreError: 353574:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.887992] LustreError: 353574:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [11409.033988] LustreError: 353556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858383 with bad export cookie 10214297402841664737 [11409.048414] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11409.058685] LustreError: 353556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11409.662639] Lustre: server umount lustre-MDT0001 complete [11425.564546] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11430.335953] Key type lgssc unregistered [11430.687879] LNet: 357117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11430.701304] LNetError: 357117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11430.750384] LNet: Removed LNI 192.168.204.146@tcp [11431.999812] Key type .llcrypt unregistered [11432.004981] Key type ._llcrypt unregistered [11465.038275] Key type ._llcrypt registered [11465.042272] Key type .llcrypt registered [11465.162867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11467.035929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11467.154634] alg: No test for adler32 (adler32-zlib) [11468.357679] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [11468.749771] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11470.527372] Key type lgssc registered [11472.018127] Lustre: Echo OBD driver; http://www.lustre.org/ [11482.675401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11482.699616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.532320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11490.996638] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11494.884086] LustreError: 358636:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.924160] LustreError: 358636:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11500.003507] LustreError: 358637:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.090427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11502.541614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11508.018189] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11517.786826] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11523.995320] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11531.075251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11531.513441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11538.821342] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11549.703476] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.061293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11569.124881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11569.140892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11569.156825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11573.884206] Lustre: server umount lustre-MDT0000 complete [11574.760150] LustreError: 358631:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.814118] LustreError: 358631:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11578.576062] LustreError: 358618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858552 with bad export cookie 10602134284893988133 [11578.596675] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11579.069308] Lustre: server umount lustre-MDT0001 complete [11594.554737] Lustre: server umount lustre-OST0000 complete [11609.902539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.544791] LustreError: 362372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.606286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11615.712671] LustreError: 362373:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.968433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11625.945461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.974837] LustreError: 362372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.974849] LustreError: 362372:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11626.551458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11631.767923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11640.861483] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11647.745736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11655.396964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.714795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.779588] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11671.325421] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11680.309943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11695.587518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.595460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11695.612526] Lustre: Skipped 3 previous similar messages [11695.636803] Lustre: Skipped 4 previous similar messages [11700.705370] LustreError: 362373:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11700.764654] Lustre: server umount lustre-MDT0000 complete [11704.783759] LustreError: 362354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858678 with bad export cookie 10602134284893989246 [11704.786901] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.804981] LustreError: 362354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11705.118493] Lustre: server umount lustre-MDT0001 complete [11719.894307] Lustre: server umount lustre-OST0000 complete [11736.404042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11736.858776] LustreError: 366306:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11736.875677] LustreError: 366306:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11736.918351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.086650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11754.950673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.642272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11769.495557] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.495890] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11784.691315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11785.040402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.046204] Lustre: Skipped 1 previous similar message [11793.069821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11803.154839] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.579436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11815.911558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11815.928094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.952577] Lustre: Skipped 1 previous similar message [11815.959137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11819.775821] Lustre: server umount lustre-OST0000 complete [11827.169881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.175112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11827.176901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.176911] Lustre: Skipped 2 previous similar messages [11827.191435] Lustre: Skipped 2 previous similar messages [11830.920588] Lustre: server umount lustre-MDT0000 complete [11832.291537] LustreError: 367035:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.306988] LustreError: 367035:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [11835.029573] LustreError: 366286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858809 with bad export cookie 10602134284893990268 [11835.040274] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11835.045683] LustreError: 366286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11835.466545] Lustre: server umount lustre-MDT0001 complete [11847.394547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11850.165942] Key type lgssc unregistered [11850.485981] LNet: 369848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11850.498174] LNetError: 369848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11850.516263] LNet: Removed LNI 192.168.204.146@tcp [11851.470728] Key type .llcrypt unregistered [11851.486332] Key type ._llcrypt unregistered [11878.401689] Key type ._llcrypt registered [11878.406334] Key type .llcrypt registered [11878.523295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11879.651949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11879.946324] alg: No test for adler32 (adler32-zlib) [11881.165470] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [11881.400474] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11883.055479] Key type lgssc registered [11884.542536] Lustre: Echo OBD driver; http://www.lustre.org/ [11893.258818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11893.326647] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.345754] LustreError: 371290:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11893.381118] LustreError: 371290:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11893.396493] LustreError: 371290:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11893.406433] LustreError: 371290:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11893.423444] LustreError: 371290:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11901.863431] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11901.868752] LustreError: 371687:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11901.885266] LustreError: 371687:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11901.893981] LustreError: 371687:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11901.899284] LustreError: 371687:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11901.905766] LustreError: 371687:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11922.733591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11926.062886] Key type lgssc unregistered [11926.343749] LNet: 372694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11926.348365] LNetError: 372694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11926.386059] LNet: Removed LNI 192.168.204.146@tcp [11927.306795] Key type .llcrypt unregistered [11927.309954] Key type ._llcrypt unregistered [11944.908433] Key type ._llcrypt registered [11944.912626] Key type .llcrypt registered [11945.010360] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11959.736776] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11961.018987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11961.184368] alg: No test for adler32 (adler32-zlib) [11962.335067] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [11962.661484] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11964.399804] Key type lgssc registered [11965.611630] Lustre: Echo OBD driver; http://www.lustre.org/ [12010.515230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.537355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.837311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12011.876716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12011.962351] Lustre: lustre-MDT0000: new disk, initializing [12012.093322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12012.141055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12018.201943] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12031.382647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.514271] Lustre: 375849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12031.576520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12031.580086] Lustre: Skipped 1 previous similar message [12031.687850] Lustre: lustre-MDT0001: new disk, initializing [12031.883386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12031.953591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12031.967119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12037.870280] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12050.517791] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12057.956783] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12066.665509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12066.939288] Lustre: lustre-OST0000: new disk, initializing [12066.941625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12066.948212] Lustre: 377821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12067.086864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12073.836632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12074.081491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12074.091859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12074.210260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12083.641078] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12088.390544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12088.658899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12093.432768] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12093.668756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12100.066907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12100.073452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12100.079789] LustreError: Skipped 1 previous similar message [12100.082146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.117587] Lustre: Skipped 1 previous similar message [12101.524669] Lustre: server umount lustre-OST0000 complete [12108.768548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12108.781362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.801030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12108.817918] Lustre: Skipped 1 previous similar message [12110.309055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12110.310633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.626625] Lustre: server umount lustre-MDT0000 complete [12115.921550] LustreError: 377820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859089 with bad export cookie 9490906138497890668 [12115.929418] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12115.935991] LustreError: 377820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12116.400725] Lustre: server umount lustre-MDT0001 complete [12127.567236] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12129.915744] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:31:42 (1787859102) [12137.127973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12137.489755] LustreError: 380284:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.513685] LustreError: 380284:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12137.575844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12142.702721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12143.082667] LustreError: 380285:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12147.181167] LustreError: 380284:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.292354] LustreError: 380285:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.521438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.038183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12159.084545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12168.973179] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12175.902981] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12183.256456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12183.689809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12191.162811] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12199.808684] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12207.947247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12208.188212] Lustre: lustre-OST0001: new disk, initializing [12208.196510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12208.203176] Lustre: 383120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12208.305361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12214.969947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12216.959373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12216.972272] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12217.057433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12225.874364] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12232.674754] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12232.689836] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.704563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12232.711668] Lustre: Skipped 2 previous similar messages [12235.919610] Lustre: server umount lustre-OST0001 complete [12237.794499] LustreError: 382107:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.817622] LustreError: 382107:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12248.039580] LustreError: 382109:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.053747] LustreError: 382109:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [12253.160340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.167437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12253.200869] Lustre: Skipped 2 previous similar messages [12253.219387] Lustre: Skipped 2 previous similar messages [12263.392792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.404097] Lustre: Skipped 3 previous similar messages [12266.975144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12267.191222] Lustre: server umount lustre-OST0000 complete [12273.633122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.360843] Lustre: server umount lustre-MDT0000 complete [12278.752734] LustreError: 380578:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12278.767473] LustreError: 380578:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [12282.804814] LustreError: 380266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859256 with bad export cookie 9490906138497891550 [12282.818678] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12282.820729] LustreError: 380266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12297.187969] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12297.722829] Lustre: server umount lustre-MDT0001 complete [12310.836193] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12314.613556] Key type lgssc unregistered [12315.078833] LNet: 385113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12315.106640] LNetError: 385113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12316.133981] LNet: Removed LNI 192.168.204.146@tcp [12317.274465] Key type .llcrypt unregistered [12317.281512] Key type ._llcrypt unregistered [12376.078598] Key type ._llcrypt registered [12376.082468] Key type .llcrypt registered [12376.261951] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12377.587262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12377.614590] alg: No test for adler32 (adler32-zlib) [12378.747846] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [12379.090710] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12380.808722] Key type lgssc registered [12382.125860] Lustre: Echo OBD driver; http://www.lustre.org/ [12390.792050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12390.831226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12392.143733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12392.176796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12392.347765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12397.027359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12405.681184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12405.777511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12405.810499] Lustre: 387916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12406.026592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12410.752591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12417.821422] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12422.775377] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12426.727656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12426.733667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.741243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12426.749382] Lustre: Skipped 1 previous similar message [12426.775876] Lustre: Skipped 1 previous similar message [12431.229763] Lustre: server umount lustre-MDT0000 complete [12434.764674] LustreError: 387907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859408 with bad export cookie 13956542419687214468 [12434.767757] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12434.773563] LustreError: 387907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12435.173304] Lustre: server umount lustre-MDT0001 complete [12445.820166] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:36:57 (1787859417) [12453.444196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12454.014218] LustreError: 390431:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.032057] LustreError: 390431:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12454.081589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12459.488207] LustreError: 390430:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.826724] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12464.613036] LustreError: 390431:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.410848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12469.735800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12469.925280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.741478] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12482.588594] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12488.923308] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12495.944797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12496.093895] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12496.308955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12502.466696] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12510.645764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12520.095062] Lustre: Permanently deactivating lustre-OST0000 [12520.118411] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12526.669466] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12528.974602] LustreError: 393368:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12528.983239] LustreError: 393368:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12528.987104] LustreError: 393368:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12532.209442] LustreError: 393416:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12532.220628] LustreError: 393416:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12540.415146] Lustre: server umount lustre-OST0000 complete [12543.978175] LustreError: 393667:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12543.982242] LustreError: 393667:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12547.041409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12547.050657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12547.088603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12548.070000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12553.185680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12553.195233] Lustre: Skipped 3 previous similar messages [12556.326329] Lustre: server umount lustre-MDT0000 complete [12558.307997] LustreError: 390426:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.334936] LustreError: 390426:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12559.552900] LustreError: 390410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859533 with bad export cookie 13956542419687215049 [12559.559203] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12559.567359] LustreError: 390410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12569.272857] Lustre: server umount lustre-MDT0001 complete [12577.947355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.358236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12584.925317] LustreError: 394826:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12591.597386] Lustre: 395165:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [12593.884578] Lustre: server umount MGS complete [12605.697624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12606.230815] LustreError: 395890:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.244479] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12606.258325] Lustre: Skipped 1 previous similar message [12606.309644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12611.303617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12616.674594] LustreError: 395891:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.694229] LustreError: 395891:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12621.979059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12622.417973] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12622.603993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12628.066800] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12638.347411] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12645.969785] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12647.587575] Lustre: Permanently reactivating lustre-OST0000 [12647.602577] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12647.606971] Lustre: Skipped 2 previous similar messages [12675.039178] LustreError: 397643:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12683.769894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12684.268483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12692.912779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12703.332465] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12709.867748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12709.881656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.903120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12712.141793] Lustre: server umount lustre-OST0000 complete [12719.584631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12719.594062] LustreError: Skipped 1 previous similar message [12719.599848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12719.606369] Lustre: Skipped 1 previous similar message [12719.610227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12719.614872] Lustre: Skipped 2 previous similar messages [12731.359290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12731.620764] Lustre: server umount lustre-MDT0000 complete [12735.961199] LustreError: 395870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859709 with bad export cookie 13956542419687216099 [12735.981279] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12735.982190] LustreError: 395870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12736.723419] Lustre: server umount lustre-MDT0001 complete [12749.819900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12752.850080] Key type lgssc unregistered [12753.111824] LNet: 400264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12753.122435] LNetError: 400264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12753.137898] LNet: Removed LNI 192.168.204.146@tcp [12754.072146] Key type .llcrypt unregistered [12754.074165] Key type ._llcrypt unregistered [12773.620751] Key type ._llcrypt registered [12773.624250] Key type .llcrypt registered [12773.852431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12788.799672] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12790.267958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12790.445908] alg: No test for adler32 (adler32-zlib) [12791.640371] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [12791.947926] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12793.712957] Key type lgssc registered [12795.057688] Lustre: Echo OBD driver; http://www.lustre.org/ [12838.151830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12838.193943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.339721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12839.369703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12839.483537] Lustre: lustre-MDT0000: new disk, initializing [12839.558375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12839.585837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12844.833450] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12855.295932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12855.407050] Lustre: 403425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12855.454887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12855.459596] Lustre: Skipped 1 previous similar message [12855.543661] Lustre: lustre-MDT0001: new disk, initializing [12855.640204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12855.676159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12855.696451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12860.264853] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12869.493025] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12875.638975] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12884.284243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12884.514240] Lustre: lustre-OST0000: new disk, initializing [12884.519203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12884.527234] Lustre: 405394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12884.649562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12893.662230] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12894.336118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12894.368882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12894.506576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12906.478883] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12911.941368] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12912.190946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12917.400418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12917.677422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12920.312852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12920.319187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12920.332159] LustreError: Skipped 1 previous similar message [12920.342337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12920.359525] Lustre: Skipped 1 previous similar message [12920.377644] Lustre: Skipped 1 previous similar message [12925.421467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12925.436831] Lustre: Skipped 1 previous similar message [12926.273496] Lustre: server umount lustre-OST0000 complete [12932.582911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12932.593124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12932.614943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12935.648843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12935.650030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.653960] Lustre: server umount lustre-MDT0000 complete [12942.360664] LustreError: 404229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859916 with bad export cookie 483677295818386710 [12942.365694] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12942.384992] LustreError: 404229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12942.879192] Lustre: server umount lustre-MDT0001 complete [12953.648752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12956.270729] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:45:28 (1787859928) [12964.216826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.238622] LustreError: 407855:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.267976] LustreError: 407855:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12965.360505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12969.966488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12970.472192] LustreError: 407856:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.589185] LustreError: 407855:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12979.656021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12980.166626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12984.736620] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12991.629794] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12996.597205] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13005.884860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13006.492056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13014.080398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13024.838555] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13027.979531] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13032.159084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13041.632702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13041.634822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.645025] LustreError: Skipped 1 previous similar message [13041.645672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13041.645676] Lustre: Skipped 1 previous similar message [13041.706984] Lustre: Skipped 1 previous similar message [13044.554681] Lustre: server umount lustre-OST0000 complete [13051.872317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13051.876757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13051.892442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13051.902795] Lustre: Skipped 1 previous similar message [13051.911995] Lustre: Skipped 1 previous similar message [13055.660232] Lustre: server umount lustre-MDT0000 complete [13056.993529] LustreError: 408584:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13057.009111] LustreError: 408584:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13059.918303] LustreError: 407835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860033 with bad export cookie 483677295818387592 [13059.929344] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13059.932863] LustreError: 407835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13060.455619] Lustre: server umount lustre-MDT0001 complete [13073.048880] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13076.110033] Key type lgssc unregistered [13076.476405] LNet: 411586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13076.490630] LNetError: 411586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13076.508694] LNet: Removed LNI 192.168.204.146@tcp [13077.423791] Key type .llcrypt unregistered [13077.426239] Key type ._llcrypt unregistered [13103.324471] Key type ._llcrypt registered [13103.327415] Key type .llcrypt registered [13103.424928] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13103.732911] Key type .llcrypt unregistered [13103.735694] Key type ._llcrypt unregistered [13121.506703] Key type ._llcrypt registered [13121.509053] Key type .llcrypt registered [13121.622323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13123.074554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13123.102344] alg: No test for adler32 (adler32-zlib) [13124.229674] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [13124.567655] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13126.215407] Key type lgssc registered [13127.643809] Lustre: Echo OBD driver; http://www.lustre.org/ [13137.740539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13137.787903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13139.499690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13144.716882] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13149.669492] LustreError: 413804:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13149.686079] LustreError: 413804:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13154.784669] LustreError: 413805:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13155.057262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13155.481163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13160.934776] LustreError: 413799:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13160.959034] LustreError: 413799:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [13163.663978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13164.022896] LustreError: 413804:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.033840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13164.037551] LustreError: 413804:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13168.037041] LustreError: 413805:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.053395] LustreError: 413805:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13172.058376] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.348404] LustreError: 413805:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.374767] LustreError: 413805:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [13179.612804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13180.093916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13184.747365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13195.237757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13195.245888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13195.261400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13200.868436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13200.883642] Lustre: Skipped 1 previous similar message [13201.357128] Lustre: server umount lustre-OST0000 complete [13206.002611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13206.024708] Lustre: Skipped 1 previous similar message [13206.033209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13206.046093] Lustre: Skipped 1 previous similar message [13208.264459] Lustre: server umount lustre-MDT0000 complete [13210.592129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13210.610458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13210.650878] LustreError: 413804:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13212.773602] LustreError: 413785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860186 with bad export cookie 17120634349943759813 [13212.778720] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13212.791761] LustreError: 413785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13213.200508] Lustre: server umount lustre-MDT0001 complete [13226.133266] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13229.432062] Key type lgssc unregistered [13229.826569] LNet: 416956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13229.832464] LNetError: 416956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13229.847719] LNet: Removed LNI 192.168.204.146@tcp [13230.788738] Key type .llcrypt unregistered [13230.794759] Key type ._llcrypt unregistered [13261.082336] Key type ._llcrypt registered [13261.084598] Key type .llcrypt registered [13261.213097] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13262.216241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13262.336808] alg: No test for adler32 (adler32-zlib) [13263.422795] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [13263.721562] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13265.383303] Key type lgssc registered [13266.672867] Lustre: Echo OBD driver; http://www.lustre.org/ [13275.622610] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13275.676341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13291.231563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13296.356096] LustreError: 418353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [13323.039194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13330.448983] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13339.164385] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13347.341194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13347.948090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13348.037351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13352.586425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13353.442243] LustreError: 418378:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.464705] LustreError: 418378:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13357.544252] LustreError: 419391:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13357.569899] LustreError: 419391:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13362.007459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13362.661954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13368.648459] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13377.665923] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13385.170878] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13393.891480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13393.905149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13393.935565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13395.812690] Lustre: server umount lustre-OST0000 complete [13400.323684] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13400.333763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13400.345071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13400.591281] Lustre: server umount lustre-MDT0000 complete [13403.619214] LustreError: 419395:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13403.619956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13405.123776] LustreError: 418359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860379 with bad export cookie 448679648784146523 [13405.125761] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13405.134732] LustreError: 418359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13415.603802] Lustre: server umount lustre-MDT0001 complete [13428.883955] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13432.213939] Key type lgssc unregistered [13432.536818] LNet: 421928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13432.549468] LNetError: 421928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13432.572979] LNet: Removed LNI 192.168.204.146@tcp [13433.576102] Key type .llcrypt unregistered [13433.580663] Key type ._llcrypt unregistered [13461.421731] Key type ._llcrypt registered [13461.430565] Key type .llcrypt registered [13461.602326] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13463.128493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13463.157268] alg: No test for adler32 (adler32-zlib) [13464.162871] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [13464.458890] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13466.183189] Key type lgssc registered [13467.683533] Lustre: Echo OBD driver; http://www.lustre.org/ [13476.999656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13477.020573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.503446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13484.724565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13488.104331] LustreError: 423348:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13488.123140] LustreError: 423348:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13493.219079] LustreError: 423347:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13495.233888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13495.771218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13501.123532] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13510.283993] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13517.938474] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13525.910603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13526.312530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13533.075980] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13542.138564] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13548.872656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13552.104032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13552.123728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13552.154398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13554.147839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13554.153489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13554.176746] Lustre: Skipped 1 previous similar message [13557.743680] Lustre: server umount lustre-MDT0000 complete [13559.264050] LustreError: 424783:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13559.291734] LustreError: 424783:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13560.781607] LustreError: 423327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.46@tcp arrived at 1787860534 with bad export cookie 4724027099299626461 [13562.773494] LustreError: 423329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860536 with bad export cookie 4724027099299625796 [13562.786452] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13562.797898] LustreError: 423329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13564.389639] LustreError: 423347:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13564.393067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.416165] LustreError: 423347:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13564.441838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13564.447796] Lustre: Skipped 3 previous similar messages [13567.393256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13569.092098] Lustre: server umount lustre-OST0000 complete [13572.515164] LustreError: 423343:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13572.530701] LustreError: 423343:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [13576.525825] Lustre: server umount lustre-MDT0001 complete [13587.769553] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13590.720251] Key type lgssc unregistered [13591.094637] LNet: 427040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13591.114744] LNetError: 427040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13591.131525] LNet: Removed LNI 192.168.204.146@tcp [13592.453133] Key type .llcrypt unregistered [13592.455412] Key type ._llcrypt unregistered [13624.465593] Key type ._llcrypt registered [13624.467162] Key type .llcrypt registered [13624.618376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13625.833843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13626.033565] alg: No test for adler32 (adler32-zlib) [13627.252977] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [13627.560427] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13629.231182] Key type lgssc registered [13631.026533] Lustre: Echo OBD driver; http://www.lustre.org/ [13640.028780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13640.062539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.721696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13647.293203] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13651.941319] LustreError: 428458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13651.957119] LustreError: 428458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13656.931302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13657.056779] LustreError: 428459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13657.473912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13662.970453] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13672.446909] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13679.689804] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13688.156688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13688.564145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13696.562779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13705.413890] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13711.098876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13716.451908] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13716.482601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13716.495764] Lustre: Skipped 1 previous similar message [13717.619022] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [13717.623590] Lustre: Skipped 1 previous similar message [13721.571057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13721.579293] Lustre: Skipped 1 previous similar message [13726.702840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13726.713511] Lustre: Skipped 2 previous similar messages [13729.247947] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13729.404887] Lustre: server umount lustre-MDT0001 complete [13731.811472] LustreError: 428459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.929114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13736.934553] LustreError: 428458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.950987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.955816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13736.955825] Lustre: Skipped 1 previous similar message [13737.036586] LustreError: 428458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13742.051684] LustreError: 428459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13742.891390] Lustre: server umount lustre-OST0000 complete [13747.697594] Lustre: server umount lustre-MDT0000 complete [13764.300303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13767.605296] Key type lgssc unregistered [13767.996818] LNet: 432149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13768.009920] LNetError: 432149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.028090] LNet: Removed LNI 192.168.204.146@tcp [13769.076830] Key type .llcrypt unregistered [13769.083580] Key type ._llcrypt unregistered [13797.622394] Key type ._llcrypt registered [13797.625487] Key type .llcrypt registered [13797.732550] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13815.284845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13815.985181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13816.019469] alg: No test for adler32 (adler32-zlib) [13816.991272] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [13817.225406] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13818.929341] Key type lgssc registered [13820.174370] Lustre: Echo OBD driver; http://www.lustre.org/ [13864.253838] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13881.998821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13882.027171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13883.275842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13883.309926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13883.392348] Lustre: lustre-MDT0000: new disk, initializing [13883.490990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13883.514067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13889.602184] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13908.303463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13908.810179] Lustre: lustre-OST0000: new disk, initializing [13908.822424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13908.829549] Lustre: Skipped 1 previous similar message [13908.835942] Lustre: 437199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13908.955438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13916.211225] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13916.817460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13916.835606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13916.856285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13926.745432] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13944.765155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13944.920699] Lustre: 436169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13945.005109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13945.090732] Lustre: lustre-MDT0001: new disk, initializing [13945.230925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13945.280263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13945.295724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13950.916640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13963.364878] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13963.715524] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13970.137224] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13970.505248] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13980.270054] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [13980.641398] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13980.650050] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13980.670091] Lustre: Skipped 1 previous similar message [13983.624544] Lustre: server umount lustre-MDT0001 complete [13990.882897] LustreError: 436175:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13990.905904] LustreError: 436175:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [13996.004876] LustreError: 439773:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13996.026618] LustreError: 439773:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14001.129215] LustreError: 439773:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14001.144635] LustreError: 439773:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14004.203847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14004.212693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14004.225591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14004.230589] Lustre: Skipped 2 previous similar messages [14006.246936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14006.264266] LustreError: 436180:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14008.094349] Lustre: server umount lustre-OST0000 complete [14011.361619] LustreError: 436181:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14018.638616] Lustre: server umount lustre-MDT0000 complete [14033.247468] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14036.033639] Key type lgssc unregistered [14036.337693] LNet: 440952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14036.343935] LNetError: 440952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14036.354495] LNet: Removed LNI 192.168.204.146@tcp [14037.373642] Key type .llcrypt unregistered [14037.377389] Key type ._llcrypt unregistered [14082.473326] Key type ._llcrypt registered [14082.476637] Key type .llcrypt registered [14082.629846] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14083.017331] Key type .llcrypt unregistered [14083.019462] Key type ._llcrypt unregistered [14101.882241] Key type ._llcrypt registered [14101.884820] Key type .llcrypt registered [14102.017968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [14102.926628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14102.959430] alg: No test for adler32 (adler32-zlib) [14104.337766] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [14104.792160] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [14106.535190] Key type lgssc registered [14108.010738] Lustre: Echo OBD driver; http://www.lustre.org/ [14138.072792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14138.102460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14139.379403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14139.431537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14139.522351] Lustre: lustre-MDT0000: new disk, initializing [14139.605190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14139.625316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14145.040571] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14157.038827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14157.182264] Lustre: 444353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14157.263122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14157.267747] Lustre: Skipped 1 previous similar message [14157.370796] Lustre: lustre-MDT0001: new disk, initializing [14157.501578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14157.562447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14157.570877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14163.046609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14173.370973] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14179.123320] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14185.953057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14186.248941] Lustre: lustre-OST0000: new disk, initializing [14186.253940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14186.269045] Lustre: 446323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14186.386979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14192.644656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14195.291348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14195.303618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14195.377068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14201.589898] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14208.997753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14209.003651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14209.030940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14211.051856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14211.056519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14211.065546] Lustre: Skipped 1 previous similar message [14214.400707] Lustre: server umount lustre-MDT0000 complete [14218.815102] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861192 with bad export cookie 2219408098307479791 [14218.827075] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14218.830779] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14219.112623] Lustre: server umount lustre-MDT0001 complete [14233.898449] Lustre: server umount lustre-OST0000 complete [14245.658829] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:06:58 (1787861218) [14255.400789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14256.108447] LustreError: 448351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14256.142483] LustreError: 448351:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [14256.220415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14261.218338] LustreError: 448352:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14261.431153] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14266.347067] LustreError: 448351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14270.916813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14271.247918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14275.846283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14283.358285] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14289.227748] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14310.440271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14310.591097] Lustre: Found index 0 for lustre-OST0000, updating log [14310.739187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14317.370490] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14326.257892] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14327.172041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14337.505343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14337.518592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14337.522586] LustreError: Skipped 1 previous similar message [14337.529624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14337.541313] Lustre: Skipped 1 previous similar message [14337.562980] Lustre: Skipped 3 previous similar messages [14342.349351] Lustre: server umount lustre-OST0000 complete [14360.036117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14360.039792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14360.049548] Lustre: Skipped 1 previous similar message [14360.070608] Lustre: Skipped 3 previous similar messages [14364.129328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14364.140804] Lustre: Skipped 1 previous similar message [14368.032356] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [14373.343126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14373.601939] Lustre: server umount lustre-MDT0000 complete [14375.395757] LustreError: 448351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14375.426309] LustreError: 448351:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14377.620514] LustreError: 450194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861351 with bad export cookie 2219408098307481184 [14377.628801] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14377.639666] LustreError: 450194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14377.953955] Lustre: server umount lustre-MDT0001 complete [14388.860967] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:09:21 (1787861361) [15018.881441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15019.214741] LustreError: 454766:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15019.287383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15022.917439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15024.611701] LustreError: 454767:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15029.729984] LustreError: 454766:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15030.551775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15030.778457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15030.786536] Lustre: Skipped 4 previous similar messages [15030.954310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15035.884302] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15043.089417] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15048.960601] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15070.715306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15071.083983] Lustre: Found index 0 for lustre-OST0000, updating log [15071.521341] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [15071.592432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15078.481667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15088.220851] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15092.718720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15092.744199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15097.109858] Lustre: server umount lustre-OST0000 complete [15102.947860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15102.964075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.986460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15104.045098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15107.173342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15107.178142] Lustre: Skipped 7 previous similar messages [15108.847393] Lustre: server umount lustre-MDT0000 complete [15109.242285] LustreError: 455126:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15109.318670] LustreError: 455126:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15113.462508] LustreError: 454766:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15113.536208] LustreError: 454766:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [15114.521696] LustreError: 454747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862088 with bad export cookie 2219408098307482269 [15114.531634] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15114.533898] LustreError: 454747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15114.888867] Lustre: server umount lustre-MDT0001 complete [15131.942344] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [15151.067956] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [15163.481838] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15211.925464] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:23:04 (1787862184) [15232.121235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15234.917823] Key type lgssc unregistered [15235.274796] LNet: 461384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15235.279597] LNetError: 461384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15332.587752] LNet: Removed LNI 192.168.204.146@tcp [15333.881341] Key type .llcrypt unregistered [15333.884551] Key type ._llcrypt unregistered [15356.245921] Key type ._llcrypt registered [15356.247375] Key type .llcrypt registered [15356.386703] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15357.654521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15357.711909] alg: No test for adler32 (adler32-zlib) [15359.058591] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [15359.445138] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [15361.151162] Key type lgssc registered [15363.117508] Lustre: Echo OBD driver; http://www.lustre.org/ [15368.889037] LNet: Added LNI 192.168.204.146@tcp100 [8/256/0/180] [15369.831740] LNet: Added LNI 192.168.204.146@tcp101 [8/256/0/180] [15370.870234] LNet: Added LNI 192.168.204.146@tcp102 [8/256/0/180] [15372.048814] LNet: Added LNI 192.168.204.146@tcp103 [8/256/0/180] [15373.148865] LNet: Added LNI 192.168.204.146@tcp104 [8/256/0/180] [15374.292880] LNet: Added LNI 192.168.204.146@tcp105 [8/256/0/180] [15375.360959] LNet: Added LNI 192.168.204.146@tcp106 [8/256/0/180] [15376.239319] LNet: Added LNI 192.168.204.146@tcp107 [8/256/0/180] [15377.399381] LNet: Added LNI 192.168.204.146@tcp108 [8/256/0/180] [15378.620706] LNet: Added LNI 192.168.204.146@tcp109 [8/256/0/180] [15379.777763] LNet: Added LNI 192.168.204.146@tcp110 [8/256/0/180] [15381.371916] LNet: Added LNI 192.168.204.146@tcp111 [8/256/0/180] [15382.892421] LNet: Added LNI 192.168.204.146@tcp112 [8/256/0/180] [15384.357813] LNet: Added LNI 192.168.204.146@tcp113 [8/256/0/180] [15385.872829] LNet: Added LNI 192.168.204.146@tcp114 [8/256/0/180] [15387.479153] LNet: Added LNI 192.168.204.146@tcp115 [8/256/0/180] [15388.882682] LNet: Added LNI 192.168.204.146@tcp116 [8/256/0/180] [15390.045541] LNet: Added LNI 192.168.204.146@tcp117 [8/256/0/180] [15391.281802] LNet: Added LNI 192.168.204.146@tcp118 [8/256/0/180] [15392.829457] LNet: Added LNI 192.168.204.146@tcp119 [8/256/0/180] [15394.086887] LNet: Added LNI 192.168.204.146@tcp120 [8/256/0/180] [15395.442238] LNet: Added LNI 192.168.204.146@tcp121 [8/256/0/180] [15396.463885] LNet: Added LNI 192.168.204.146@tcp122 [8/256/0/180] [15397.508715] LNet: Added LNI 192.168.204.146@tcp123 [8/256/0/180] [15398.361531] LNet: Added LNI 192.168.204.146@tcp124 [8/256/0/180] [15399.178592] LNet: Added LNI 192.168.204.146@tcp125 [8/256/0/180] [15400.234810] LNet: Added LNI 192.168.204.146@tcp126 [8/256/0/180] [15401.090445] LNet: Added LNI 192.168.204.146@tcp127 [8/256/0/180] [15402.129749] LNet: Added LNI 192.168.204.146@tcp128 [8/256/0/180] [15403.106965] LNet: Added LNI 192.168.204.146@tcp129 [8/256/0/180] [15404.083368] LNet: Added LNI 192.168.204.146@tcp130 [8/256/0/180] [15405.439513] LNet: Added LNI 192.168.204.146@tcp131 [8/256/0/180] [15406.449713] LNet: Added LNI 192.168.204.146@tcp132 [8/256/0/180] [15407.619599] LNet: Added LNI 192.168.204.146@tcp133 [8/256/0/180] [15408.626462] LNet: Added LNI 192.168.204.146@tcp134 [8/256/0/180] [15409.570615] LNet: Added LNI 192.168.204.146@tcp135 [8/256/0/180] [15416.870583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15416.907093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15418.112116] Lustre: 464417:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [15418.206313] LNetError: 464438:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15418.227666] LNetError: Protocol error connecting to 192.168.204.146@tcp114 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15418.244876] LNetError: Skipped 4 previous similar messages [15418.325882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15418.349186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15418.419238] Lustre: lustre-MDT0000: new disk, initializing [15418.490089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15418.505747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15418.847247] LNet: 5 local NIs in recovery (showing 5): 192.168.204.146@tcp126, 192.168.204.146@tcp127, 192.168.204.146@tcp128, 192.168.204.146@tcp129, 192.168.204.146@tcp125 [15419.233334] LNetError: 461877:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15419.246072] LNetError: 461877:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15419.255110] LNetError: Protocol error connecting to 192.168.204.146@tcp120 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15419.276870] LNetError: Skipped 1 previous similar message [15421.925354] LNetError: 464438:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15421.939720] LNetError: 464438:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15421.957790] LNetError: Protocol error connecting to 192.168.204.146@tcp127 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15421.981099] LNetError: Skipped 5 previous similar messages [15424.598696] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15426.031797] LNetError: 461877:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15426.068569] LNetError: Protocol error connecting to 192.168.204.146@tcp127 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15434.220269] LNetError: 464445:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15434.230641] LNetError: Protocol error connecting to 192.168.204.146@tcp127 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15437.457040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15437.690458] Lustre: 464433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15437.790483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15437.799020] Lustre: Skipped 1 previous similar message [15437.986523] Lustre: lustre-MDT0001: new disk, initializing [15438.076448] Lustre: 465238:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [15438.092688] Lustre: 465238:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15438.382583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15438.452553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15438.470771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15442.669950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15447.168741] Lustre: 465638:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [15447.186706] Lustre: 465638:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15450.592340] LNetError: 461874:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15450.604693] LNetError: Protocol error connecting to 192.168.204.146@tcp127 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15454.899943] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15462.872986] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15468.077278] LNetError: 464438:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15468.105290] LNetError: 464438:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15468.117618] LNetError: Protocol error connecting to 192.168.204.146@tcp121 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15468.128633] LNetError: Skipped 7 previous similar messages [15472.004586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15472.548822] Lustre: lustre-OST0000: new disk, initializing [15472.552931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15472.560166] Lustre: 466410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15472.776115] Lustre: 466416:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [15472.785291] Lustre: 466416:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15472.853168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15478.239885] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp127 [15479.362765] Lustre: 466695:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [15479.377034] Lustre: 466695:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15479.440119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15479.454648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15479.730544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15481.200745] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15493.175702] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15501.856561] LNetError: 464445:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [15501.872837] LNetError: 464445:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15501.888882] LNetError: Protocol error connecting to 192.168.204.146@tcp120 at host 192.168.204.146:988: is it running a compatible version of Lustre? [15501.905687] LNetError: Skipped 12 previous similar messages [15505.379222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15505.381641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15505.409466] LustreError: Skipped 1 previous similar message [15505.412554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15505.436454] Lustre: Skipped 1 previous similar message [15505.478759] Lustre: Skipped 1 previous similar message [15508.969895] Lustre: server umount lustre-OST0000 complete [15515.107491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15515.115846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15515.138835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15519.858392] Lustre: server umount lustre-MDT0000 complete [15524.285427] LustreError: 464425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862498 with bad export cookie 17862564498228274411 [15524.295454] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15524.300165] LustreError: 464425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15524.794381] Lustre: server umount lustre-MDT0001 complete [15539.386066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15542.770488] Key type lgssc unregistered [15543.201042] LNet: 468329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15543.209869] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15543.218871] LNet: Removed LNI 192.168.204.146@tcp [15544.287294] LNet: Removed LNI 192.168.204.146@tcp100 [15544.295539] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15544.305697] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15544.322475] LNet: Removed LNI 192.168.204.146@tcp101 [15545.375860] LNet: Removed LNI 192.168.204.146@tcp102 [15545.379983] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15545.393359] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15545.402280] LNet: Removed LNI 192.168.204.146@tcp103 [15546.463371] LNet: Removed LNI 192.168.204.146@tcp104 [15547.551502] LNet: Removed LNI 192.168.204.146@tcp105 [15547.562939] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15547.575356] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15547.595730] LNet: Removed LNI 192.168.204.146@tcp106 [15548.641285] LNet: Removed LNI 192.168.204.146@tcp107 [15549.663174] LNet: Removed LNI 192.168.204.146@tcp108 [15550.687478] LNet: Removed LNI 192.168.204.146@tcp109 [15551.711883] LNet: Removed LNI 192.168.204.146@tcp110 [15551.724200] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15551.734852] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15551.750851] LNet: Removed LNI 192.168.204.146@tcp111 [15552.799202] LNet: Removed LNI 192.168.204.146@tcp112 [15553.823200] LNet: Removed LNI 192.168.204.146@tcp113 [15554.847257] LNet: Removed LNI 192.168.204.146@tcp114 [15555.871664] LNet: Removed LNI 192.168.204.146@tcp115 [15556.895203] LNet: Removed LNI 192.168.204.146@tcp116 [15557.919488] LNet: Removed LNI 192.168.204.146@tcp117 [15558.943594] LNet: Removed LNI 192.168.204.146@tcp118 [15559.967791] LNet: Removed LNI 192.168.204.146@tcp119 [15559.987087] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15559.998349] LNetError: 468329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15560.017215] LNet: Removed LNI 192.168.204.146@tcp120 [15561.055567] LNet: Removed LNI 192.168.204.146@tcp121 [15562.143221] LNet: Removed LNI 192.168.204.146@tcp122 [15563.167614] LNet: Removed LNI 192.168.204.146@tcp123 [15564.191607] LNet: Removed LNI 192.168.204.146@tcp124 [15565.279736] LNet: Removed LNI 192.168.204.146@tcp125 [15566.303664] LNet: Removed LNI 192.168.204.146@tcp126 [15566.308691] LNet: Removed LNI 192.168.204.146@tcp127 [15567.327278] LNet: Removed LNI 192.168.204.146@tcp128 [15568.351316] LNet: Removed LNI 192.168.204.146@tcp129 [15568.352994] LNet: Removed LNI 192.168.204.146@tcp130 [15568.357295] LNet: Removed LNI 192.168.204.146@tcp131 [15568.368441] LNet: Removed LNI 192.168.204.146@tcp132 [15568.377827] LNet: Removed LNI 192.168.204.146@tcp133 [15568.395130] LNet: Removed LNI 192.168.204.146@tcp134 [15568.417538] LNet: Removed LNI 192.168.204.146@tcp135 [15569.653986] Key type .llcrypt unregistered [15569.656092] Key type ._llcrypt unregistered [15598.467572] Key type ._llcrypt registered [15598.469940] Key type .llcrypt registered [15598.602189] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [15615.075507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15616.415132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15616.562330] alg: No test for adler32 (adler32-zlib) [15617.704482] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [15618.005470] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [15619.687166] Key type lgssc registered [15621.077641] Lustre: Echo OBD driver; http://www.lustre.org/ [15660.699979] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:30:32 (1787862632) [15681.253706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [15685.010294] Key type lgssc unregistered [15685.396990] LNet: 472423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15685.422462] LNetError: 472423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15685.463884] LNet: Removed LNI 192.168.204.146@tcp [15686.677505] Key type .llcrypt unregistered [15686.679255] Key type ._llcrypt unregistered [15705.127366] Key type ._llcrypt registered [15705.128529] Key type .llcrypt registered [15705.317728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [15706.314870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15706.361150] alg: No test for adler32 (adler32-zlib) [15707.702477] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [15708.057679] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [15709.815395] Key type lgssc registered [15711.235190] Lustre: Echo OBD driver; http://www.lustre.org/ [15721.820991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15721.858150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15723.207693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15723.268642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15723.352583] Lustre: lustre-MDT0000: new disk, initializing [15723.497899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15723.532402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15729.232947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15740.348287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15740.460809] Lustre: 473640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15740.512981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15740.518120] Lustre: Skipped 1 previous similar message [15740.576336] Lustre: lustre-MDT0001: new disk, initializing [15740.650873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15740.706082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15740.725330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15746.839761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15758.810860] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15765.102160] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15773.150497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15773.603678] Lustre: lustre-OST0000: new disk, initializing [15773.606864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15773.614210] Lustre: 475609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15773.705522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15779.422986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15779.431581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15779.586624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15780.906431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [15791.703511] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15797.278830] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lnet_if_list [15804.374633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15804.421432] LNet: Added LNI 192.168.204.146@tcp10 [8/256/0/180] [15812.071991] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15812.108734] LNet: Added LNI 192.168.204.146@tcp11 [8/256/0/180] [15816.635266] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15816.682444] LNet: Added LNI 192.168.204.146@tcp12 [8/256/0/180] [15821.619636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15821.650007] LNet: Added LNI 192.168.204.146@tcp13 [8/256/0/180] [15826.810833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15826.841374] LNet: Added LNI 192.168.204.146@tcp14 [8/256/0/180] [15833.308444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15833.361822] LNet: Added LNI 192.168.204.146@tcp15 [8/256/0/180] [15838.944476] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15838.967587] LNet: Added LNI 192.168.204.146@tcp16 [8/256/0/180] [15844.477257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15844.505645] LNet: Added LNI 192.168.204.146@tcp17 [8/256/0/180] [15850.393772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15850.429844] LNet: Added LNI 192.168.204.146@tcp18 [8/256/0/180] [15857.100995] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15857.125850] LNet: Added LNI 192.168.204.146@tcp19 [8/256/0/180] [15863.290966] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15863.309750] LNet: Added LNI 192.168.204.146@tcp20 [8/256/0/180] [15869.230616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15869.294133] LNet: Added LNI 192.168.204.146@tcp21 [8/256/0/180] [15875.646790] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15875.687954] LNet: Added LNI 192.168.204.146@tcp22 [8/256/0/180] [15882.559637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15882.593791] LNet: Added LNI 192.168.204.146@tcp23 [8/256/0/180] [15889.233154] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15889.292799] LNet: Added LNI 192.168.204.146@tcp24 [8/256/0/180] [15896.436953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15896.482867] LNet: Added LNI 192.168.204.146@tcp25 [8/256/0/180] [15905.028165] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15905.070237] LNet: Added LNI 192.168.204.146@tcp26 [8/256/0/180] [15914.419711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15914.444386] LNet: Added LNI 192.168.204.146@tcp27 [8/256/0/180] [15920.770717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15920.849761] LNet: Added LNI 192.168.204.146@tcp28 [8/256/0/180] [15930.615960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15930.664949] LNet: Added LNI 192.168.204.146@tcp29 [8/256/0/180] [15940.380575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15940.401035] LNet: Added LNI 192.168.204.146@tcp30 [8/256/0/180] [15948.632870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15948.663481] LNet: Added LNI 192.168.204.146@tcp31 [8/256/0/180] [15955.976816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15956.016920] LNet: Added LNI 192.168.204.146@tcp32 [8/256/0/180] [15962.775609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15962.829424] LNet: Added LNI 192.168.204.146@tcp33 [8/256/0/180] [15969.839923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15969.889058] LNet: Added LNI 192.168.204.146@tcp34 [8/256/0/180] [15977.390674] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15977.417891] LNet: Added LNI 192.168.204.146@tcp35 [8/256/0/180] [15986.238845] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15986.291356] LNet: Added LNI 192.168.204.146@tcp36 [8/256/0/180] [15995.129514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15995.165082] LNet: Added LNI 192.168.204.146@tcp37 [8/256/0/180] [16003.175069] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16003.219537] LNet: Added LNI 192.168.204.146@tcp38 [8/256/0/180] [16011.210321] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16011.258904] LNet: Added LNI 192.168.204.146@tcp39 [8/256/0/180] [16018.986613] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16019.019485] LNet: Added LNI 192.168.204.146@tcp40 [8/256/0/180] [16026.500191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16026.522962] LNet: Added LNI 192.168.204.146@tcp41 [8/256/0/180] [16034.717761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16034.742370] LNet: Added LNI 192.168.204.146@tcp42 [8/256/0/180] [16043.297701] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16043.328672] LNet: Added LNI 192.168.204.146@tcp43 [8/256/0/180] [16051.979103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16051.999168] LNet: Added LNI 192.168.204.146@tcp44 [8/256/0/180] [16059.945916] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16059.989072] LNet: Added LNI 192.168.204.146@tcp45 [8/256/0/180] [16068.667427] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16068.708502] LNet: Added LNI 192.168.204.146@tcp46 [8/256/0/180] [16078.072663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16078.162396] LNet: Added LNI 192.168.204.146@tcp47 [8/256/0/180] [16089.533427] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16089.602364] LNet: Added LNI 192.168.204.146@tcp48 [8/256/0/180] [16100.844961] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16100.874541] LNet: Added LNI 192.168.204.146@tcp49 [8/256/0/180] [16110.777726] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16110.801604] LNet: Added LNI 192.168.204.146@tcp50 [8/256/0/180] [16133.105422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16133.117321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16133.140977] Lustre: Skipped 2 previous similar messages [16133.151948] Lustre: Skipped 2 previous similar messages [16138.231657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16138.250468] Lustre: Skipped 2 previous similar messages [16138.711557] Lustre: server umount lustre-MDT0000 complete [16144.641486] LustreError: 473632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863118 with bad export cookie 17991493270060639127 [16144.651054] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16144.654774] LustreError: 473632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16145.443139] Lustre: server umount lustre-MDT0001 complete [16164.831161] Lustre: 472912:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863122/real 1787863122] req@000000007637aba8 x1874709899828608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863138 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16164.860415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16165.855342] Lustre: 472910:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863123/real 1787863123] req@00000000c447c292 x1874709899828736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16169.238547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [16170.015104] Lustre: 472910:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863127/real 1787863127] req@00000000f026ac89 x1874709899828864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16175.071125] Lustre: 472912:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863133/real 1787863133] req@000000004a32d717 x1874709899829248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16175.108642] Lustre: 472912:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16179.504167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16180.192505] Lustre: 472909:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863138/real 1787863138] req@000000001be74e96 x1874709899829376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787863154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16190.498785] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf9ae9fce5ccbad7d [16190.508377] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [16190.791771] LustreError: 475603:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16190.830593] LustreError: 475603:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [16190.915205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16194.912262] LustreError: 475604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16196.029316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16199.012244] LustreError: 475603:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16199.017377] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16204.268554] LustreError: 475604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16206.378559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16206.746687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16209.851903] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16212.447450] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16220.741787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16228.203640] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16250.855352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16250.875964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16250.881959] Lustre: Skipped 1 previous similar message [16254.492410] Lustre: server umount lustre-OST0000 complete [16261.109649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16261.122996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16261.133325] Lustre: Skipped 2 previous similar messages [16261.164906] Lustre: Skipped 1 previous similar message [16264.535608] Lustre: server umount lustre-MDT0000 complete [16266.213377] LustreError: 499866:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16266.240740] LustreError: 499866:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [16268.747133] LustreError: 473630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863242 with bad export cookie 17991493270060641661 [16268.760704] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16268.763304] LustreError: 473630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16269.205292] Lustre: server umount lustre-MDT0001 complete [16280.652557] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [16283.759190] Key type lgssc unregistered [16284.153391] LNet: 502667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16284.160315] LNetError: 502667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16284.166937] LNet: Removed LNI 192.168.204.146@tcp [16284.170297] LNet: Removed LNI 192.168.204.146@tcp10 [16284.174253] LNet: Removed LNI 192.168.204.146@tcp11 [16284.180144] LNet: Removed LNI 192.168.204.146@tcp12 [16284.182385] LNet: Removed LNI 192.168.204.146@tcp13 [16284.188540] LNet: Removed LNI 192.168.204.146@tcp14 [16284.192875] LNet: Removed LNI 192.168.204.146@tcp15 [16284.196378] LNet: Removed LNI 192.168.204.146@tcp16 [16284.201510] LNet: Removed LNI 192.168.204.146@tcp17 [16284.205506] LNet: Removed LNI 192.168.204.146@tcp18 [16284.209661] LNet: Removed LNI 192.168.204.146@tcp19 [16284.213536] LNet: Removed LNI 192.168.204.146@tcp20 [16284.215665] LNet: Removed LNI 192.168.204.146@tcp21 [16284.219770] LNet: Removed LNI 192.168.204.146@tcp22 [16284.223305] LNet: Removed LNI 192.168.204.146@tcp23 [16284.227353] LNet: Removed LNI 192.168.204.146@tcp24 [16284.230458] LNet: Removed LNI 192.168.204.146@tcp25 [16284.233443] LNet: Removed LNI 192.168.204.146@tcp26 [16284.235624] LNet: Removed LNI 192.168.204.146@tcp27 [16284.238296] LNet: Removed LNI 192.168.204.146@tcp28 [16284.241237] LNet: Removed LNI 192.168.204.146@tcp29 [16284.244585] LNet: Removed LNI 192.168.204.146@tcp30 [16284.250375] LNet: Removed LNI 192.168.204.146@tcp31 [16284.255078] LNet: Removed LNI 192.168.204.146@tcp32 [16284.259958] LNet: Removed LNI 192.168.204.146@tcp33 [16284.263399] LNet: Removed LNI 192.168.204.146@tcp34 [16284.266631] LNet: Removed LNI 192.168.204.146@tcp35 [16284.268644] LNet: Removed LNI 192.168.204.146@tcp36 [16284.271354] LNet: Removed LNI 192.168.204.146@tcp37 [16284.275461] LNet: Removed LNI 192.168.204.146@tcp38 [16284.278333] LNet: Removed LNI 192.168.204.146@tcp39 [16284.281549] LNet: Removed LNI 192.168.204.146@tcp40 [16284.284731] LNet: Removed LNI 192.168.204.146@tcp41 [16284.287297] LNet: Removed LNI 192.168.204.146@tcp42 [16284.289870] LNet: Removed LNI 192.168.204.146@tcp43 [16284.293251] LNet: Removed LNI 192.168.204.146@tcp44 [16284.296519] LNet: Removed LNI 192.168.204.146@tcp45 [16284.299307] LNet: Removed LNI 192.168.204.146@tcp46 [16284.301848] LNet: Removed LNI 192.168.204.146@tcp47 [16284.304393] LNet: Removed LNI 192.168.204.146@tcp48 [16284.308570] LNet: Removed LNI 192.168.204.146@tcp49 [16284.321431] LNet: Removed LNI 192.168.204.146@tcp50 [16285.177414] Key type .llcrypt unregistered [16285.179052] Key type ._llcrypt unregistered [16306.520389] Key type ._llcrypt registered [16306.523584] Key type .llcrypt registered [16306.636545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [16321.695728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [16322.917799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16322.985083] alg: No test for adler32 (adler32-zlib) [16324.219534] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [16324.474519] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [16326.143189] Key type lgssc registered [16327.750000] Lustre: Echo OBD driver; http://www.lustre.org/ [16368.388471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16368.407125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16369.715564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16369.745626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16369.815485] Lustre: lustre-MDT0000: new disk, initializing [16369.901348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16369.917996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16374.742319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16385.866598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16385.977901] Lustre: 505865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16386.058144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16386.063400] Lustre: Skipped 1 previous similar message [16386.173853] Lustre: lustre-MDT0001: new disk, initializing [16386.363044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16386.457330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16386.466231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16390.655916] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16400.355166] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16406.663764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16415.642537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16415.998697] Lustre: *** cfs_fail_loc=90f, val=0*** [16416.122602] Lustre: lustre-OST0000: new disk, initializing [16416.127577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16416.145721] Lustre: 507880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16416.292826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16422.979715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16423.005678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16423.198823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16424.113317] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16434.680667] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16443.881299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16443.884252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16443.888735] LustreError: Skipped 1 previous similar message [16443.891763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16443.903927] Lustre: Skipped 1 previous similar message [16443.926995] Lustre: Skipped 1 previous similar message [16448.342461] Lustre: server umount lustre-OST0000 complete [16453.097696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16453.107543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16453.127048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16454.114717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16454.128989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16454.133413] Lustre: Skipped 1 previous similar message [16458.910323] Lustre: server umount lustre-MDT0000 complete [16463.023544] LustreError: 507879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863437 with bad export cookie 10867346565639352564 [16463.031771] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16463.038276] LustreError: 507879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16463.548966] Lustre: server umount lustre-MDT0001 complete [16473.733068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [16476.381422] Key type lgssc unregistered [16476.699659] LNet: 509791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16476.706931] LNetError: 509791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16476.738866] LNet: Removed LNI 192.168.204.146@tcp [16477.565863] Key type .llcrypt unregistered [16477.569636] Key type ._llcrypt unregistered [16497.861655] Key type ._llcrypt registered [16497.863773] Key type .llcrypt registered [16497.965379] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [16499.279541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16499.331408] alg: No test for adler32 (adler32-zlib) [16500.307126] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [16500.514424] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [16502.199279] Key type lgssc registered [16504.038153] Lustre: Echo OBD driver; http://www.lustre.org/ [16513.093041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16513.106456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16514.624706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16519.552375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16524.774360] LustreError: 511064:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16524.789664] LustreError: 511064:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [16528.461672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16528.784500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16533.680910] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16541.809683] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16547.857777] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16555.459117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16555.792192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16562.666246] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16569.891137] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16584.619723] loop0: detected capacity change from 0 to 204800000 [16584.826024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16584.846106] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16584.858299] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16591.607926] loop0: detected capacity change from 0 to 204800000 [16591.874762] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16591.883576] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16591.893432] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16598.232883] loop0: detected capacity change from 0 to 204800000 [16604.310684] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16604.535100] Lustre: fs15246-OST0000: new disk, initializing [16604.545164] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16604.552160] Lustre: 514657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16604.639661] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16611.486068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16621.058503] loop1: detected capacity change from 0 to 204800000 [16626.534136] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16626.614786] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16626.671061] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16626.756448] Lustre: fs15246-MDT0000: new disk, initializing [16626.906121] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16626.928486] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16627.032673] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16628.065223] Lustre: 515782:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f10aa554 x1874710730161408/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787863618 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16631.495437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16633.137669] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16633.154420] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16633.208917] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16653.294649] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16653.313912] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16657.400489] Lustre: server umount fs15246-MDT0000 complete [16658.406607] LustreError: 513556:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16663.526669] LustreError: 511060:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16666.408590] Lustre: server umount fs15246-OST0000 complete [16677.856097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16677.860887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16677.873344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16678.883900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16678.895105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16678.918850] Lustre: Skipped 1 previous similar message [16682.766863] Lustre: server umount lustre-OST0000 complete [16687.586060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16687.597572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16687.616841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16694.244452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16694.252727] Lustre: Skipped 3 previous similar messages [16700.383364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16700.548599] Lustre: server umount lustre-MDT0000 complete [16703.923782] LustreError: 511044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863677 with bad export cookie 7631561845232305808 [16703.925352] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16703.933982] LustreError: 511044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16704.281387] Lustre: server umount lustre-MDT0001 complete [16715.424667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [16718.488229] Key type lgssc unregistered [16718.851834] LNet: 518881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16718.859627] LNetError: 518881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16718.881727] LNet: Removed LNI 192.168.204.146@tcp [16719.784930] Key type .llcrypt unregistered [16719.789973] Key type ._llcrypt unregistered [16737.841467] Key type ._llcrypt registered [16737.850637] Key type .llcrypt registered [16737.988367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [16739.172230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16739.198601] alg: No test for adler32 (adler32-zlib) [16740.413424] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [16740.753651] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [16742.449050] Key type lgssc registered [16743.613109] Lustre: Echo OBD driver; http://www.lustre.org/ [16783.414262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [16794.086037] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [16829.450677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16829.492479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16830.700880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16830.747637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16830.853402] Lustre: lustre-MDT0000: new disk, initializing [16830.941697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16830.965928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16835.641333] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16848.680286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16848.836505] Lustre: 523350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16848.912886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16848.916971] Lustre: Skipped 1 previous similar message [16848.994231] Lustre: lustre-MDT0001: new disk, initializing [16849.098471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16849.145201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16849.176227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16853.906785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16864.532239] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16870.856602] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16878.719689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16879.169772] Lustre: lustre-OST0000: new disk, initializing [16879.173613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16879.185966] Lustre: 525320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16879.300529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16886.365803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16886.392263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16886.528528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16886.749209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [16898.518612] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16903.865715] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16904.108738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16908.889726] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16909.110986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16912.356169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16912.361276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16912.367625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16912.372769] Lustre: Skipped 1 previous similar message [16912.396237] LustreError: Skipped 1 previous similar message [16917.481288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16917.491478] Lustre: Skipped 2 previous similar messages [16918.107492] Lustre: server umount lustre-OST0000 complete [16926.179405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16926.195320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16926.248324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16927.725428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16929.897404] Lustre: server umount lustre-MDT0000 complete [16934.150582] LustreError: 523341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863908 with bad export cookie 668421540125382568 [16934.158203] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16934.166238] LustreError: 523341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16934.399814] Lustre: server umount lustre-MDT0001 complete [16963.466198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [16978.682913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [17017.739549] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16704 sec ========= 16:53:09 (1787863989) [17019.839421] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:53:12 (1787863992) === [17023.813334] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:53:16 (1787863996) === [17064.947430] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [17067.811309] Key type lgssc unregistered [17068.114868] LNet: 531991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17068.122172] LNetError: 531991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17068.141441] LNet: Removed LNI 192.168.204.146@tcp [17068.984800] Key type .llcrypt unregistered [17068.988522] Key type ._llcrypt unregistered