[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 822194929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003054] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008382] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011129] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.022637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027523] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031022] *** VALIDATE tmpfs *** [ 0.032447] *** VALIDATE proc *** [ 0.034113] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043103] debug: unmapping init [mem 0xffffffff86459000-0xffffffff86460fff] [ 0.045262] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047019] ... version: 2 [ 0.048008] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052010] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.054281] rcu: Hierarchical SRCU implementation. [ 0.056349] smp: Bringing up secondary CPUs ... [ 0.057602] x86: Booting SMP configuration: [ 0.058021] .... node #0, CPUs: #1 #2 #3 [ 0.066197] smp: Brought up 1 node, 4 CPUs [ 0.068009] smpboot: Max logical packages: 1 [ 0.069022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.097025] node 0 deferred pages initialised in 26ms [ 0.100227] devtmpfs: initialized [ 0.101241] x86/mm: Memory block size: 128MB [ 0.103845] gcov: version magic: 0x41383552 [ 0.105276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107224] pinctrl core: initialized pinctrl subsystem [ 0.108172] [ 0.108809] ************************************************************* [ 0.109012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110015] ** ** [ 0.111012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112011] ** ** [ 0.113013] ** This means that this kernel is built to expose internal ** [ 0.114013] ** IOMMU data structures, which may compromise security on ** [ 0.115013] ** your system. ** [ 0.116014] ** ** [ 0.117012] ** If you see this message and you are not debugging the ** [ 0.118011] ** kernel, report this immediately to your vendor! ** [ 0.119015] ** ** [ 0.120012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121011] ************************************************************* [ 0.122724] NET: Registered protocol family 16 [ 0.123533] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.127009] cpuidle: using governor menu [ 0.130568] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137076] PCI: Using configuration type 1 for base access [ 0.141672] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.154389] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.157060] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.160475] cryptd: max_cpu_qlen set to 1000 [ 0.161815] ACPI: Added _OSI(Module Device) [ 0.162000] ACPI: Added _OSI(Processor Device) [ 0.164013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.171100] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.179160] ACPI: Interpreter enabled [ 0.180064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.181015] ACPI: Using IOAPIC for interrupt routing [ 0.182193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.183511] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196321] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.197060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.199078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.201885] acpiphp: Slot [2] registered [ 0.202125] acpiphp: Slot [5] registered [ 0.203107] acpiphp: Slot [6] registered [ 0.204150] acpiphp: Slot [7] registered [ 0.205097] acpiphp: Slot [8] registered [ 0.206142] acpiphp: Slot [9] registered [ 0.207137] acpiphp: Slot [10] registered [ 0.208114] acpiphp: Slot [3] registered [ 0.209131] acpiphp: Slot [4] registered [ 0.210088] acpiphp: Slot [11] registered [ 0.211078] acpiphp: Slot [12] registered [ 0.213077] acpiphp: Slot [13] registered [ 0.214241] acpiphp: Slot [14] registered [ 0.216115] acpiphp: Slot [15] registered [ 0.218100] acpiphp: Slot [16] registered [ 0.219073] acpiphp: Slot [17] registered [ 0.220086] acpiphp: Slot [18] registered [ 0.222109] acpiphp: Slot [19] registered [ 0.223000] acpiphp: Slot [20] registered [ 0.223000] acpiphp: Slot [21] registered [ 0.226100] acpiphp: Slot [22] registered [ 0.227138] acpiphp: Slot [23] registered [ 0.229082] acpiphp: Slot [24] registered [ 0.230137] acpiphp: Slot [25] registered [ 0.232098] acpiphp: Slot [26] registered [ 0.233103] acpiphp: Slot [27] registered [ 0.235134] acpiphp: Slot [28] registered [ 0.236137] acpiphp: Slot [29] registered [ 0.238111] acpiphp: Slot [30] registered [ 0.239221] acpiphp: Slot [31] registered [ 0.241113] PCI host bridge to bus 0000:00 [ 0.242034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245078] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.247021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.249031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.256028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.261158] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.265412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.282734] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.288015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.291017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294411] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.296084] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300813] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.319014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.324013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.331105] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.350018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.435963] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489046] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.514015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528042] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.566018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.584281] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.606015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.616017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.641017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.656000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.667083] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.685035] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.722017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740903] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.760015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.774015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.806025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.824114] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.826351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.829629] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.832617] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.835489] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.842121] iommu: Default domain type: Passthrough [ 0.844415] SCSI subsystem initialized [ 0.846132] ACPI: bus type USB registered [ 0.848116] usbcore: registered new interface driver usbfs [ 0.850094] usbcore: registered new interface driver hub [ 0.852150] usbcore: registered new device driver usb [ 0.854204] pps_core: LinuxPPS API ver. 1 registered [ 0.856011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.860064] PTP clock support registered [ 0.862041] EDAC MC: Ver: 3.0.0 [ 0.864276] PCI: Using ACPI for IRQ routing [ 0.865994] NetLabel: Initializing [ 0.867016] NetLabel: domain hash size = 128 [ 0.869011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.871086] NetLabel: unlabeled traffic allowed by default [ 0.874446] vgaarb: loaded [ 0.876468] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.878018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.884883] clocksource: Switched to clocksource kvm-clock [ 1.003446] VFS: Disk quotas dquot_6.6.0 [ 1.004746] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.006615] *** VALIDATE ramfs *** [ 1.007711] *** VALIDATE hugetlbfs *** [ 1.009037] pnp: PnP ACPI init [ 1.011220] pnp: PnP ACPI: found 6 devices [ 1.028256] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.030867] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.032992] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.034849] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.036936] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.038971] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.041557] NET: Registered protocol family 2 [ 1.043712] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.048551] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.051940] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.056434] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.059352] TCP: Hash tables configured (established 65536 bind 65536) [ 1.061665] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.064293] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.066496] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.068921] NET: Registered protocol family 1 [ 1.071207] RPC: Registered named UNIX socket transport module. [ 1.072930] RPC: Registered udp transport module. [ 1.074555] RPC: Registered tcp transport module. [ 1.075966] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.077885] NET: Registered protocol family 44 [ 1.079538] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.084327] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.086084] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.087967] PCI: CLS 0 bytes, default 64 [ 1.089479] Unpacking initramfs... [ 2.639539] debug: unmapping init [mem 0xffff9a3b7cc54000-0xffff9a3b7ffbffff] [ 2.643674] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.645935] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.649030] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.159174] Initialise system trusted keyrings [ 3.160752] Key type blacklist registered [ 3.162503] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.171491] zbud: loaded [ 3.175117] *** VALIDATE nfs *** [ 3.176395] *** VALIDATE nfs4 *** [ 3.178232] pstore: using deflate compression [ 3.183563] Platform Keyring initialized [ 3.309512] NET: Registered protocol family 38 [ 3.311419] Key type asymmetric registered [ 3.312943] Asymmetric key parser 'x509' registered [ 3.315303] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.318933] io scheduler mq-deadline registered [ 3.320666] io scheduler kyber registered [ 3.322059] io scheduler bfq registered [ 3.325175] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.328814] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.332265] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.335389] ACPI: Power Button [PWRF] [ 3.341442] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.348894] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.380713] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.396538] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.429240] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.459180] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.488459] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.496183] Non-volatile memory driver v1.3 [ 3.497698] Linux agpgart interface v0.103 [ 3.534472] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.537722] vda: detected capacity change from 0 to 74862592 [ 3.555959] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.559519] vdb: detected capacity change from 0 to 1073741824 [ 3.592599] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.595645] vdc: detected capacity change from 0 to 2621440000 [ 3.626319] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.629964] vdd: detected capacity change from 0 to 2621440000 [ 3.661334] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.663977] vde: detected capacity change from 0 to 4294967296 [ 3.692501] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.695213] vdf: detected capacity change from 0 to 4294967296 [ 3.714509] libphy: Fixed MDIO Bus: probed [ 3.732061] usbcore: registered new interface driver usbserial_generic [ 3.734652] usbserial: USB Serial support registered for generic [ 3.737392] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.742502] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.744440] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.747344] mousedev: PS/2 mouse device common for all mice [ 3.751227] rtc_cmos 00:05: RTC can wake from S4 [ 3.756758] rtc_cmos 00:05: registered as rtc0 [ 3.758817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.760775] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.762156] intel_pstate: CPU model not supported [ 3.765194] hid: raw HID events driver (C) Jiri Kosina [ 3.771205] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.775306] usbcore: registered new interface driver usbhid [ 3.779000] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.781959] usbhid: USB HID core driver [ 3.782231] drop_monitor: Initializing network drop monitor service [ 3.782370] Initializing XFRM netlink socket [ 3.782749] NET: Registered protocol family 10 [ 3.790545] Segment Routing with IPv6 [ 3.799235] NET: Registered protocol family 17 [ 3.801446] mpls_gso: MPLS GSO support [ 3.806548] RAS: Correctable Errors collector initialized. [ 3.810293] AVX version of gcm_enc/dec engaged. [ 3.812258] AES CTR mode by8 optimization enabled [ 3.896736] sched_clock: Marking stable (3896660605, 0)->(5033289954, -1136629349) [ 3.900599] registered taskstats version 1 [ 3.902673] Loading compiled-in X.509 certificates [ 3.904585] zswap: loaded using pool lzo/zbud [ 3.928578] Key type big_key registered [ 3.940865] Key type encrypted registered [ 3.942585] ima: No TPM chip found, activating TPM-bypass! [ 3.944747] ima: Allocated hash algorithm: sha1 [ 3.946370] ima: No architecture policies found [ 3.948278] evm: Initialising EVM extended attributes: [ 3.950182] evm: security.selinux [ 3.951388] evm: security.ima [ 3.952549] evm: security.capability [ 3.953848] evm: HMAC attrs: 0x1 [ 3.956356] rtc_cmos 00:05: setting system clock to 2026-08-24 05:10:11 UTC (1787548211) [ 3.962840] debug: unmapping init [mem 0xffffffff87403000-0xffffffff875fffff] [ 3.965955] debug: unmapping init [mem 0xffffffff86182000-0xffffffff86458fff] [ 3.976068] Write protecting the kernel read-only data: 28672k [ 3.978325] debug: unmapping init [mem 0xffffffff84803000-0xffffffff849fffff] [ 3.980069] debug: unmapping init [mem 0xffffffff85114000-0xffffffff851fffff] [ 4.016182] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.025516] systemd[1]: Detected virtualization kvm. [ 4.027526] systemd[1]: Detected architecture x86-64. [ 4.029555] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.059562] systemd[1]: No hostname configured. [ 4.061379] systemd[1]: Set hostname to . [ 4.063751] random: systemd: uninitialized urandom read (16 bytes read) [ 4.066360] systemd[1]: Initializing machine ID from random generator. [ 4.127388] random: ln: uninitialized urandom read (6 bytes read) [ 4.207671] random: systemd: uninitialized urandom read (16 bytes read) [ 4.210173] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.216781] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.227519] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.040391] device-mapper: uevent: version 1.0.3 [ 5.042908] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.927699] virtio_net virtio0 ens2: renamed from eth0 [ 5.994160] random: fast init done [ 6.158399] scsi host0: ata_piix [ 6.177187] scsi host1: ata_piix [ 6.183372] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.189985] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.470658] random: crng init done [ 10.473020] random: 7 urandom warning(s) missed due to ratelimiting [ 14.126217] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.915829] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.151553] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.331408] SELinux: Disabled at runtime. [ 20.502467] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.520426] systemd[1]: Detected virtualization kvm. [ 20.522297] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.355559] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.363279] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.374499] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.395778] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.405670] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.458515] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.472432] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 23.008609] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.353369] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.924208] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.012928] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.506479] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.568235] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 32.587869] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 33.258297] NFS: Registering the id_resolver key type [ 33.269686] Key type id_resolver registered [ 33.289249] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg127-server login: [ 49.355115] hrtimer: interrupt took 3883881 ns [ 106.209304] libcfs: loading out-of-tree module taints kernel. [ 106.266264] Key type ._llcrypt registered [ 106.269417] Key type .llcrypt registered [ 106.461610] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 127.152829] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 128.777273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.785516] alg: No test for adler32 (adler32-zlib) [ 130.196893] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 131.003693] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 132.791310] Key type lgssc registered [ 134.649755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 155.059890] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.984171] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 221.034364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 221.069207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.337170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.366330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.458886] Lustre: lustre-MDT0000: new disk, initializing [ 222.581293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.596831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.867759] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.145158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.327362] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 242.354378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 242.363620] Lustre: Skipped 1 previous similar message [ 242.468583] Lustre: lustre-MDT0001: new disk, initializing [ 242.568528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.623607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 242.636665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.217439] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.824258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 264.967868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.302680] Lustre: lustre-OST0000: new disk, initializing [ 265.306792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 265.312987] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.370400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 267.936623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 267.951140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 268.031323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 272.987376] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.328781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.639710] Lustre: lustre-OST0001: new disk, initializing [ 293.642321] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 293.653961] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 293.811067] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 299.106090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 299.123302] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 299.234391] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 302.849852] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 316.455180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 324.401323] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 332.581873] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing check_logdir /tmp/testlogs/ [ 339.163937] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing yml_node [ 343.992690] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 346.627087] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 349.181962] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 351.002711] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 01:15:58 EDT 2026 [ 370.826556] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 372.377579] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 391.158602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 391.166734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.177821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.888973] Lustre: server umount lustre-MDT0000 complete [ 398.645209] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548606 with bad export cookie 13924607017786755435 [ 398.651217] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.661943] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 401.889094] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.892296] LustreError: 9961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.911782] Lustre: Skipped 3 previous similar messages [ 401.912489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 401.935915] LustreError: 9961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 401.974355] Lustre: Skipped 4 previous similar messages [ 403.950329] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.952431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 403.986975] Lustre: Skipped 1 previous similar message [ 405.081232] Lustre: server umount lustre-MDT0001 complete [ 409.697596] Lustre: server umount lustre-OST0000 complete [ 413.906545] Lustre: server umount lustre-OST0001 complete [ 421.215931] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 430.938516] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 472.412757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.779530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.839475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.956409] Lustre: lustre-MDT0000: new disk, initializing [ 473.105373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.133794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 478.592133] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 490.614825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.768870] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 490.871783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 490.878134] Lustre: Skipped 1 previous similar message [ 490.998860] Lustre: lustre-MDT0001: new disk, initializing [ 491.124870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 491.180452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 491.197635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 496.087230] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 507.731191] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 516.258572] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.549786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.720841] Lustre: lustre-OST0000: new disk, initializing [ 525.723199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 525.727588] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 525.789510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.420276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 527.438260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 527.561572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 533.649490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 548.542705] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.852462] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 555.674871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 561.278513] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 561.492702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 563.688277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.688660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.694625] LustreError: Skipped 1 previous similar message [ 563.697629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.716661] Lustre: Skipped 2 previous similar messages [ 568.807293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.822452] Lustre: Skipped 2 previous similar messages [ 570.251804] Lustre: server umount lustre-OST0000 complete [ 578.024524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 578.050390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.083593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 581.708810] Lustre: server umount lustre-MDT0000 complete [ 584.162036] LustreError: 15095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.185395] LustreError: 15095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 585.833494] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548793 with bad export cookie 13924607017786756961 [ 585.851203] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.215525] Lustre: server umount lustre-MDT0001 complete [ 590.785537] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:19:57 (1787548797) [ 593.983270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 596.241907] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:20:03 (1787548803) [ 605.002255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.900296] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.141799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.152295] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 611.298793] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.324917] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 620.392379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.736499] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 636.769897] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 644.596862] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 655.545173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.156659] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 674.601810] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 687.076588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 687.086692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.098232] Lustre: Skipped 1 previous similar message [ 687.103408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.108919] Lustre: Skipped 2 previous similar messages [ 691.171497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 693.472645] Lustre: server umount lustre-OST0000 complete [ 701.409590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.427727] Lustre: Skipped 2 previous similar messages [ 705.754824] Lustre: server umount lustre-MDT0000 complete [ 706.533250] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.555674] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 710.208464] LustreError: 19446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548917 with bad export cookie 13924607017786757850 [ 710.215663] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.237812] LustreError: 19446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 711.032859] Lustre: server umount lustre-MDT0001 complete [ 723.390946] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 727.090205] Key type lgssc unregistered [ 727.705589] LNet: 23030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.712405] LNetError: 23030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.745320] LNet: Removed LNI 192.168.201.127@tcp [ 729.646697] Key type .llcrypt unregistered [ 729.650054] Key type ._llcrypt unregistered [ 761.157559] Key type ._llcrypt registered [ 761.164735] Key type .llcrypt registered [ 761.302738] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 763.057726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 763.186029] alg: No test for adler32 (adler32-zlib) [ 764.388651] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 764.776127] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 766.617250] Key type lgssc registered [ 768.228663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.860128] loop: module loaded [ 777.977239] loop0: detected capacity change from 0 to 204800000 [ 778.357720] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.367404] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.877659] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 786.023667] loop0: detected capacity change from 0 to 204800000 [ 786.209517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 786.232981] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 786.248968] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 792.689399] loop0: detected capacity change from 0 to 204800000 [ 799.383446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 799.403819] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.597276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 800.641236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 800.702481] Lustre: lustre-MDT0000: new disk, initializing [ 800.784903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.824351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 805.336941] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 815.381215] loop1: detected capacity change from 0 to 204800000 [ 821.657614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.303325] Lustre: lustre-OST07c6: new disk, initializing [ 822.311226] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 822.319297] Lustre: Skipped 1 previous similar message [ 822.334644] Lustre: 26217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 822.457444] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 827.612358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 827.626294] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 827.686609] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 829.666594] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 835.664991] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 835.674801] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 835.682323] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 835.689742] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 835.694172] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 835.698382] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 852.556740] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 852.576177] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 852.587441] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 852.593798] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 852.599950] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 852.603222] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 857.935537] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 857.945628] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 857.951717] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 857.957264] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.962631] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 857.967422] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.979462] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 857.985796] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.995395] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 858.003666] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 858.014167] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 858.021452] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.284962] Lustre: 26823:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 861.300605] Lustre: 26823:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 861.309510] Lustre: 26823:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 861.315525] Lustre: 26823:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.323760] Lustre: 26823:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 861.330962] Lustre: 26823:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.343140] Lustre: 26823:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 861.353545] Lustre: 26823:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.360043] Lustre: 26823:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 861.367114] Lustre: 26823:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.381180] Lustre: 26823:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 861.394664] Lustre: 26823:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 866.441555] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 866.450923] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 866.456744] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 866.460556] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 866.463846] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 866.469654] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 866.472944] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 866.484688] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 866.490620] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 866.499209] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 866.511542] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 866.519657] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.931735] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 874.948630] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 874.957930] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 874.972439] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 874.979728] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 874.986292] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 874.994541] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 875.000732] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 875.005800] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 875.010396] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 875.016743] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 875.021969] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 935.399343] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.403478] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.414258] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 938.277116] Lustre: server umount lustre-OST07c6 complete [ 949.247033] Lustre: server umount lustre-MDT0000 complete [ 967.244345] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 970.915399] Key type lgssc unregistered [ 971.177860] LNet: 28784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.185936] LNetError: 28784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.265424] LNet: Removed LNI 192.168.201.127@tcp [ 972.977396] Key type .llcrypt unregistered [ 972.979381] Key type ._llcrypt unregistered [ 991.428381] Key type ._llcrypt registered [ 991.430979] Key type .llcrypt registered [ 991.519197] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 992.336875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.371559] alg: No test for adler32 (adler32-zlib) [ 993.549879] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 993.862373] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 995.559194] Key type lgssc registered [ 996.760622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1005.510330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1005.528930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.917502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.743568] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1017.317652] LustreError: 30053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.345353] LustreError: 30053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1021.634921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.978923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.976191] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1040.435626] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1048.392905] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1058.630049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.288076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.240107] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1076.217461] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1076.735256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1097.188945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1097.191682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.195075] LustreError: Skipped 1 previous similar message [ 1097.207403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.215525] Lustre: Skipped 1 previous similar message [ 1100.950219] Lustre: server umount lustre-OST0000 complete [ 1107.434736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.450297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.454357] Lustre: Skipped 2 previous similar messages [ 1110.953978] Lustre: server umount lustre-MDT0000 complete [ 1112.553741] LustreError: 30053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.573575] LustreError: 30053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1114.975046] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549322 with bad export cookie 7979652460066952143 [ 1114.989506] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.992525] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1115.452249] Lustre: server umount lustre-MDT0001 complete [ 1127.649515] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1132.376823] Key type lgssc unregistered [ 1132.824623] LNet: 33794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.842255] LNetError: 33794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.897906] LNet: Removed LNI 192.168.201.127@tcp [ 1135.530740] Key type .llcrypt unregistered [ 1135.538487] Key type ._llcrypt unregistered [ 1157.007964] Key type ._llcrypt registered [ 1157.012217] Key type .llcrypt registered [ 1157.122280] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1158.468726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.535951] alg: No test for adler32 (adler32-zlib) [ 1159.586464] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1159.852257] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1161.570302] Key type lgssc registered [ 1162.496987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.747494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.774669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.373896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.869726] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1183.714569] LustreError: 35067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.742994] LustreError: 35067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1188.838594] LustreError: 35068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.442915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.772648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1196.376298] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1204.949113] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1210.250736] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.289941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.972259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.013946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1224.082373] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1232.563113] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.763188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1249.771430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.782301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.811304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.821064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.827651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.847449] Lustre: Skipped 1 previous similar message [ 1254.225652] Lustre: server umount lustre-OST0000 complete [ 1262.051844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.072274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.095046] Lustre: Skipped 1 previous similar message [ 1265.059762] Lustre: server umount lustre-MDT0000 complete [ 1267.172428] LustreError: 35796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.187452] LustreError: 35796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1270.463423] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549478 with bad export cookie 11739898032561677535 [ 1270.486118] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.491703] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1271.212791] Lustre: server umount lustre-MDT0001 complete [ 1285.349735] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1288.278803] Key type lgssc unregistered [ 1288.576627] LNet: 38626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.584749] LNetError: 38626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.601819] LNet: Removed LNI 192.168.201.127@tcp [ 1289.416145] Key type .llcrypt unregistered [ 1289.423574] Key type ._llcrypt unregistered [ 1314.183309] Key type ._llcrypt registered [ 1314.185364] Key type .llcrypt registered [ 1314.321529] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1315.816323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.887246] alg: No test for adler32 (adler32-zlib) [ 1316.946577] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1317.201925] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1318.911602] Key type lgssc registered [ 1319.807917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1327.204754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1327.242275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.971788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.982217] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1339.362568] LustreError: 40043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.401788] LustreError: 40043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1344.482733] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.910425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.378558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1351.462408] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1359.161337] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1365.622504] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1373.748547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.113933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.158492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1381.127808] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1390.670219] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1398.755955] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1400.802230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.812478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.848828] Lustre: Skipped 1 previous similar message [ 1403.843517] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1403.854036] Lustre: Skipped 2 previous similar messages [ 1404.447479] Lustre: server umount lustre-MDT0000 complete [ 1405.924355] LustreError: 40040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.956169] LustreError: 40040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1408.338836] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549615 with bad export cookie 4029646279471550158 [ 1408.340387] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.345580] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.987271] LustreError: 40039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.997663] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 1411.041050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.053601] Lustre: Skipped 1 previous similar message [ 1413.093726] LustreError: 40038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.093992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1413.118577] LustreError: 40038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1413.135704] Lustre: Skipped 1 previous similar message [ 1414.728794] Lustre: server umount lustre-MDT0001 complete [ 1436.458741] Lustre: server umount lustre-OST0000 complete [ 1458.390763] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1461.584307] Key type lgssc unregistered [ 1461.983705] LNet: 43912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.990238] LNetError: 43912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.006512] LNet: Removed LNI 192.168.201.127@tcp [ 1463.128885] Key type .llcrypt unregistered [ 1463.134362] Key type ._llcrypt unregistered [ 1486.501358] Key type ._llcrypt registered [ 1486.503425] Key type .llcrypt registered [ 1486.605438] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1487.578603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.597229] alg: No test for adler32 (adler32-zlib) [ 1488.855556] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1489.087055] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1490.767157] Key type lgssc registered [ 1491.596674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1499.802723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1499.824609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.245941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.827442] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1511.394088] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.417388] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1514.802587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.389426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1520.836344] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1529.285919] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1537.132574] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1544.554887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1544.837343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.918010] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1550.621187] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1559.565663] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.698067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.714069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.728313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.741736] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1570.795719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1570.806789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1570.807603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.752507] Lustre: server umount lustre-OST0000 complete [ 1590.565019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.568201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.577451] Lustre: Skipped 1 previous similar message [ 1590.581345] Lustre: Skipped 1 previous similar message [ 1596.897764] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.911656] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1596.966520] Lustre: server umount lustre-MDT0000 complete [ 1600.851881] LustreError: 47190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549808 with bad export cookie 13741030384266459909 [ 1600.861969] LustreError: 47190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1600.868899] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.327595] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1615.491174] Lustre: server umount lustre-MDT0001 complete [ 1626.237046] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1629.894384] Key type lgssc unregistered [ 1630.307106] LNet: 49060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.318060] LNetError: 49060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.404334] LNet: Removed LNI 192.168.201.127@tcp [ 1632.533700] Key type .llcrypt unregistered [ 1632.535879] Key type ._llcrypt unregistered [ 1657.706212] Key type ._llcrypt registered [ 1657.709436] Key type .llcrypt registered [ 1657.839566] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1658.953097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1658.979793] alg: No test for adler32 (adler32-zlib) [ 1659.959884] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1660.182324] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1661.871505] Key type lgssc registered [ 1663.433911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1672.774408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1672.809497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.583552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.689377] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1684.961149] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.973995] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1690.082061] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.890363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.406268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.220927] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1707.448799] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.610831] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1722.860792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1723.277264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.048611] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1743.556935] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.821778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1751.117410] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1756.395650] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1769.252521] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 1h39m35s [ 1770.680501] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1774.564927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.568355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.586479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.649658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.658118] Lustre: Skipped 4 previous similar messages [ 1777.123460] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1781.732914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.743041] Lustre: Skipped 3 previous similar messages [ 1787.889748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.909575] Lustre: Skipped 3 previous similar messages [ 1788.384678] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1788.698133] Lustre: server umount lustre-MDT0000 complete [ 1790.401289] LustreError: 51580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.024316] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.060308] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1794.149764] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1800.164697] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.175430] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1803.213078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.414480] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.742747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.820782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1804.810608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1810.331989] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1815.699067] Lustre: DEBUG MARKER: Wait for df (36778) ... [ 1817.721826] Lustre: DEBUG MARKER: done [ 1823.713801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.719830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.738745] Lustre: Skipped 2 previous similar messages [ 1823.749655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.177823] Lustre: server umount lustre-OST0000 complete [ 1835.491760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1835.496432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.510401] LustreError: Skipped 1 previous similar message [ 1835.521997] Lustre: Skipped 1 previous similar message [ 1839.881981] Lustre: server umount lustre-MDT0000 complete [ 1842.665719] LustreError: 50473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.708073] LustreError: 50473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1844.994552] LustreError: 50457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550052 with bad export cookie 10516837789710909423 [ 1845.006674] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.392533] Lustre: server umount lustre-MDT0001 complete [ 1860.269562] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1863.362896] Key type lgssc unregistered [ 1863.576939] LNet: 55409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.580906] LNetError: 55409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.612875] LNet: Removed LNI 192.168.201.127@tcp [ 1865.328457] Key type .llcrypt unregistered [ 1865.330512] Key type ._llcrypt unregistered [ 1922.408381] Key type ._llcrypt registered [ 1922.417069] Key type .llcrypt registered [ 1922.657549] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1924.424675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.656095] alg: No test for adler32 (adler32-zlib) [ 1925.830846] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1926.003086] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1927.639252] Key type lgssc registered [ 1929.689072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1940.170148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1940.204404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.694461] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1941.767071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1942.019881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.175760] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1957.060155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.198728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1957.284214] Lustre: 57626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.665949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.518342] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1972.690455] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.592274] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.910931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.193490] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1986.514270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.502868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1995.522672] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2005.594265] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.369425] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 1h43m40s [ 2014.976481] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2023.186940] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2026.293568] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2026.405978] LustreError: 57632:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2046.948872] Lustre: lustre-MDT0000: Client 3f5f5969-5786-4570-a26c-b74060ede565 (at 192.168.201.27@tcp) reconnecting [ 2066.436124] LustreError: 57632:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2069.297416] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2071.617366] Lustre: DEBUG MARKER: done [ 2081.251152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2081.263740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.285911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.217737] Lustre: server umount lustre-OST0000 complete [ 2090.982180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.992314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.011342] Lustre: Skipped 1 previous similar message [ 2091.037567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.051301] Lustre: Skipped 1 previous similar message [ 2092.134068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.149559] Lustre: Skipped 2 previous similar messages [ 2094.731917] LustreError: 61204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000002948bd89) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2094.953398] Lustre: server umount lustre-MDT0000 complete [ 2099.467446] LustreError: 57617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550307 with bad export cookie 5761447790143623779 [ 2099.472981] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.477047] LustreError: 57617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2099.957795] Lustre: server umount lustre-MDT0001 complete [ 2112.006421] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2115.345506] Key type lgssc unregistered [ 2115.750650] LNet: 61958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.759668] LNetError: 61958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.809696] LNet: Removed LNI 192.168.201.127@tcp [ 2116.906462] Key type .llcrypt unregistered [ 2116.908438] Key type ._llcrypt unregistered [ 2168.861672] Key type ._llcrypt registered [ 2168.868739] Key type .llcrypt registered [ 2169.140826] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2172.064055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.072477] alg: No test for adler32 (adler32-zlib) [ 2173.355262] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2173.565412] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2175.185053] Key type lgssc registered [ 2176.147880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2181.961696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2181.987747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.375467] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2183.501555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2183.836320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.300848] Lustre: Failing over lustre-MDT0000 [ 2185.592487] Lustre: server umount lustre-MDT0000 complete [ 2194.075128] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:46:40 (1787550400) [ 2204.414247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.146823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.052425] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2228.397529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.702501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2228.785851] Lustre: 64762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2229.151324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.601157] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2247.591496] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2254.526447] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.176693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.360347] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2263.805435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.483404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2273.506366] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2285.746167] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2293.614276] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2302.440234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.448308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.463264] Lustre: Skipped 1 previous similar message [ 2302.484322] Lustre: Skipped 2 previous similar messages [ 2307.555488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.562803] Lustre: Skipped 2 previous similar messages [ 2308.538428] Lustre: server umount lustre-MDT0000 complete [ 2312.649786] LustreError: 64752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550520 with bad export cookie 14626024619086506658 [ 2312.655912] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.661205] LustreError: 64752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2312.706297] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.729261] Lustre: Skipped 1 previous similar message [ 2312.745971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2312.987436] Lustre: server umount lustre-MDT0001 complete [ 2317.107099] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2322.354898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2332.485385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.274672] LustreError: 63679:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c78f56a6 x1874382428354048/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2338.821481] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.848704] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2338.946646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.031668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2343.011313] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.012800] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2344.704247] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2348.514747] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.613394] LustreError: 68536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.641366] LustreError: 68536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2354.273163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.871413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2360.153622] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2360.302291] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2371.242235] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.395225] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.496592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2390.514972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.557670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.030845] Lustre: server umount lustre-MDT0000 complete [ 2396.131363] LustreError: 68537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.157936] LustreError: 68537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2399.008190] LustreError: 64752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550606 with bad export cookie 14626024619086508933 [ 2399.028759] LustreError: 64752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2399.032454] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.482649] Lustre: server umount lustre-MDT0001 complete [ 2405.936162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2416.189840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.608049] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550608/real 1787550608] req@0000000088b37040 x1874382428382464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2416.632158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.650663] Lustre: Skipped 2 previous similar messages [ 2420.448458] Lustre: 63682:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550611/real 1787550611] req@000000002b38d496 x1874382428382592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2421.727319] Lustre: 63680:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550613/real 1787550613] req@00000000ac3914fa x1874382428382720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2423.840894] LustreError: 63679:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ac3914fa x1874382428383488/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2424.286437] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.384682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.495990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2425.439110] Lustre: 63681:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550616/real 1787550616] req@00000000cd49f305 x1874382428382976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2427.939857] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2431.462920] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550622/real 1787550622] req@00000000d71a6447 x1874382428383104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2436.484525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.216348] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2442.298706] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2446.413380] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2452.258565] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2460.896734] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2471.905265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.910651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.934852] Lustre: Skipped 2 previous similar messages [ 2471.955751] Lustre: Skipped 5 previous similar messages [ 2474.371824] Lustre: server umount lustre-MDT0000 complete [ 2477.030328] LustreError: 71220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.066519] LustreError: 71220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2479.366298] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550686 with bad export cookie 14626024619086510312 [ 2479.368718] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.381859] LustreError: 64754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2479.693799] Lustre: server umount lustre-MDT0001 complete [ 2487.126360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2491.547312] Lustre: DEBUG MARKER: files compared the same [ 2504.222856] Lustre: server umount lustre-OST0000 complete [ 2523.769262] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2527.932502] Key type lgssc unregistered [ 2528.281909] LNet: 74501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.293927] LNetError: 74501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.339696] LNet: Removed LNI 192.168.201.127@tcp [ 2529.573741] Key type .llcrypt unregistered [ 2529.576987] Key type ._llcrypt unregistered [ 2554.233889] Key type ._llcrypt registered [ 2554.236825] Key type .llcrypt registered [ 2554.347233] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2555.135906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.169334] alg: No test for adler32 (adler32-zlib) [ 2556.167715] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2556.419523] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2558.071714] Key type lgssc registered [ 2559.403019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2567.234677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2567.297405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.926555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.921812] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2579.433346] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.442762] LustreError: 75921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2583.396729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.783531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.666564] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2600.278863] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2608.600810] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2616.888887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.408703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.571430] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2635.190733] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2638.916654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2654.180210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.190818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.197297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2654.200104] Lustre: Skipped 1 previous similar message [ 2659.300418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.308170] Lustre: Skipped 1 previous similar message [ 2664.416730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.423957] Lustre: Skipped 1 previous similar message [ 2666.975728] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2667.220639] Lustre: server umount lustre-OST0000 complete [ 2671.072682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2671.080166] LustreError: Skipped 1 previous similar message [ 2671.082691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.093041] Lustre: Skipped 1 previous similar message [ 2671.096612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.657159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.689281] Lustre: server umount lustre-MDT0000 complete [ 2679.776729] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.808212] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2682.627076] LustreError: 75901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550890 with bad export cookie 15445140159560040236 [ 2682.643590] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.189641] Lustre: server umount lustre-MDT0001 complete [ 2699.769977] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2704.466752] Key type lgssc unregistered [ 2704.769776] LNet: 79480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2704.785261] LNetError: 79480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2704.805353] LNet: Removed LNI 192.168.201.127@tcp [ 2706.004158] Key type .llcrypt unregistered [ 2706.011881] Key type ._llcrypt unregistered [ 2740.160252] Key type ._llcrypt registered [ 2740.162608] Key type .llcrypt registered [ 2740.276651] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2741.511538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2741.811422] alg: No test for adler32 (adler32-zlib) [ 2743.006785] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2743.328531] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2745.015591] Key type lgssc registered [ 2746.234661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.824480] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2755.881528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.487888] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2776.607298] LustreError: 80874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [ 2803.295154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.864835] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2824.468869] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2833.576388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.722690] Lustre: *** cfs_fail_loc=706, val=0*** [ 2834.063822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.171451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2838.920754] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2843.618569] LustreError: 80900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.646672] LustreError: 80900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2848.741928] LustreError: 80899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.772343] LustreError: 80899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2850.275768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.631604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.986925] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2863.837977] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2870.318300] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2875.360289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2875.374164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.418080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.387262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.405557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.413205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.527274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2881.541599] Lustre: Skipped 1 previous similar message [ 2886.643141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.649820] Lustre: Skipped 1 previous similar message [ 2887.135275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2887.649047] Lustre: server umount lustre-OST0000 complete [ 2896.871270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.871873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.882336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.883041] Lustre: Skipped 1 previous similar message [ 2896.902168] Lustre: Skipped 1 previous similar message [ 2899.607880] Lustre: server umount lustre-MDT0000 complete [ 2902.003235] LustreError: 81965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.035958] LustreError: 81965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2904.581510] LustreError: 80882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551112 with bad export cookie 13381927716387216249 [ 2904.589775] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.607289] LustreError: 80882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.041920] Lustre: server umount lustre-MDT0001 complete [ 2916.917770] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2920.370821] Key type lgssc unregistered [ 2920.616766] LNet: 84495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.626413] LNetError: 84495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.640858] LNet: Removed LNI 192.168.201.127@tcp [ 2921.666420] Key type .llcrypt unregistered [ 2921.671824] Key type ._llcrypt unregistered [ 2950.439967] Key type ._llcrypt registered [ 2950.442840] Key type .llcrypt registered [ 2950.596227] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2952.119837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2952.235496] alg: No test for adler32 (adler32-zlib) [ 2953.336408] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2953.522807] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2955.215841] Key type lgssc registered [ 2957.135518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2966.353961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2966.393846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.648811] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2982.378664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.060336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.514703] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.569316] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2990.681358] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2992.617247] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.659932] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2997.742601] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.794199] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3002.851041] LustreError: 86598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.199715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.862933] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.910150] LustreError: 86600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3006.937661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.836876] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3018.231369] LustreError: 86603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.276042] LustreError: 86603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3028.268300] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.296606] Lustre: MGS: Client 15c894a0-ac0a-4c8f-b466-89b302e09a60 (at 0@lo) reconnecting [ 3028.323216] LustreError: 86614:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3028.451443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3028.457858] Lustre: Skipped 2 previous similar messages [ 3028.661284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.805684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3034.751382] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3055.083695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.089897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.113354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.196391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.205795] Lustre: Skipped 1 previous similar message [ 3060.643576] Lustre: server umount lustre-OST0000 complete [ 3070.436240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.443819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.477223] Lustre: Skipped 2 previous similar messages [ 3070.517520] Lustre: Skipped 2 previous similar messages [ 3072.920913] Lustre: server umount lustre-MDT0000 complete [ 3075.660172] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.670885] LustreError: 86604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3078.456125] Lustre: server umount lustre-MDT0001 complete [ 3084.330641] Lustre: server umount MGS complete [ 3098.915739] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:01:45 (1787551305) [ 3131.298332] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3147.975900] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3190.939891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.411382] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3213.331178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.790831] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3213.882690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3214.066996] Lustre: lustre-MDT0001: new disk, initializing [ 3214.224555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.274768] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3214.294498] Lustre: 93836:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3214.315889] LustreError: 93836:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3220.207619] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3236.137699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.512545] Lustre: lustre-OST0000: new disk, initializing [ 3236.519166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3236.530830] Lustre: 94603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3236.625441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.168838] Lustre: 94852:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000699e870f x1874383246242944/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787551466 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3244.738175] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3257.752392] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.462981] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3265.676988] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.708881] Lustre: MGS: Client 37cecc9a-bc91-4eca-ac96-b017f023b28b (at 0@lo) reconnecting [ 3265.806301] Lustre: 95650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3265.835732] LustreError: 93822:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3265.849868] LustreError: 93822:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3265.877531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3265.991819] Lustre: lustre-MDT0000: new disk, initializing [ 3266.277487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.306399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3267.361890] Lustre: 95674:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ed8aa177 x1874383246250368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787551490 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3272.137215] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3272.480778] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3276.522599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3276.540144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3276.607760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3299.304030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.310996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.314447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.325663] Lustre: Skipped 1 previous similar message [ 3304.710930] Lustre: server umount lustre-OST0000 complete [ 3309.539327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.539897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3309.541446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.541452] Lustre: Skipped 3 previous similar messages [ 3309.558794] Lustre: Skipped 1 previous similar message [ 3323.359181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3323.496168] Lustre: server umount lustre-MDT0000 complete [ 3324.901535] LustreError: 95977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.303189] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3342.531825] Lustre: server umount lustre-MDT0001 complete [ 3346.925499] Lustre: server umount MGS complete [ 3358.843522] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:06:05 (1787551565) [ 3383.864462] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3387.052950] Key type lgssc unregistered [ 3387.414966] LNet: 98178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.428124] LNetError: 98178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3387.453969] LNet: Removed LNI 192.168.201.127@tcp [ 3388.527777] Key type .llcrypt unregistered [ 3388.529463] Key type ._llcrypt unregistered [ 3412.587505] Key type ._llcrypt registered [ 3412.593805] Key type .llcrypt registered [ 3412.867172] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3415.198328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.658464] alg: No test for adler32 (adler32-zlib) [ 3417.347779] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3417.875273] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3419.799223] Key type lgssc registered [ 3421.815591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3436.786832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3436.892354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.899741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.835205] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3448.497405] LustreError: 99403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.519331] LustreError: 99403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3453.924257] LustreError: 99402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.493393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.021753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3464.713728] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3475.345667] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3484.236698] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3493.164446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3493.970918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.391941] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3509.414325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3516.997156] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3528.674522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.689272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.715315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.217746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3530.221751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.234672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.265031] Lustre: Skipped 1 previous similar message [ 3534.510541] Lustre: server umount lustre-OST0000 complete [ 3539.937055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3539.947721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.976674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.576209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.588882] Lustre: Skipped 3 previous similar messages [ 3550.688860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.699633] Lustre: Skipped 1 previous similar message [ 3554.271159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3554.574747] Lustre: server umount lustre-MDT0000 complete [ 3554.795409] LustreError: 99403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.826233] LustreError: 99403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3559.917705] LustreError: 99398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.967590] LustreError: 99398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3560.426630] LustreError: 99383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551767 with bad export cookie 11716200065596634983 [ 3560.431945] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.434676] LustreError: 99383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3561.143073] Lustre: server umount lustre-MDT0001 complete [ 3576.606694] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3580.112625] Key type lgssc unregistered [ 3580.421890] LNet: 102946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.442234] LNetError: 102946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.478238] LNet: Removed LNI 192.168.201.127@tcp [ 3582.824629] Key type .llcrypt unregistered [ 3582.826704] Key type ._llcrypt unregistered [ 3603.322050] Key type ._llcrypt registered [ 3603.323719] Key type .llcrypt registered [ 3603.464561] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3605.011067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3605.067074] alg: No test for adler32 (adler32-zlib) [ 3606.170243] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3606.431592] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3608.167958] Key type lgssc registered [ 3609.704366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3616.751457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3616.762631] LustreError: 104032:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3616.785463] LustreError: 104033:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3616.797309] LustreError: 104032:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3616.808340] LustreError: 104032:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3616.818813] LustreError: 104032:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3616.841474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.629152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.030356] LustreError: 104070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.062948] LustreError: 104070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3630.431104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.146427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3638.466360] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3645.963903] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing lsmod [ 3649.820635] LustreError: 105237:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3649.874479] LustreError: 105239:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3649.887347] LustreError: 105237:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4953 [ 3649.903729] LustreError: 105237:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3649.908609] LustreError: 105237:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3649.921577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3650.723454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.846639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.868520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.878080] Lustre: Skipped 2 previous similar messages [ 3655.910985] Lustre: Skipped 2 previous similar messages [ 3661.256637] Lustre: server umount lustre-MDT0000 complete [ 3661.291856] LustreError: 104070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.322032] LustreError: 104070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3665.852873] LustreError: 104046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551873 with bad export cookie 14223134931119905671 [ 3665.865452] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.867815] LustreError: 104046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3666.167282] LustreError: 104502:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e0df5b98 x1874383929890176/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3666.472226] Lustre: server umount lustre-MDT0001 complete [ 3681.260216] Lustre: server umount lustre-OST0000 complete [ 3693.137239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.802682] LustreError: 106305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.935953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.085396] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3699.183827] LustreError: 106304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.298989] LustreError: 106305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.556279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.035273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3716.628170] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3725.134655] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3734.976096] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3745.106842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.949388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.602586] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3764.402551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3767.358969] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.993763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3777.002562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.044710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.065468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.070888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.083990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.126491] Lustre: Skipped 1 previous similar message [ 3785.186838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3785.201738] Lustre: Skipped 1 previous similar message [ 3790.307849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.317519] Lustre: Skipped 1 previous similar message [ 3791.327664] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3791.861794] Lustre: server umount lustre-OST0000 complete [ 3797.985721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3798.003880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.550811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.563794] Lustre: Skipped 2 previous similar messages [ 3803.998058] Lustre: server umount lustre-MDT0000 complete [ 3805.666044] LustreError: 106299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.691063] LustreError: 106299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3808.541417] LustreError: 106284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552016 with bad export cookie 14223134931119906371 [ 3808.556422] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.557791] LustreError: 106284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3809.106738] Lustre: server umount lustre-MDT0001 complete [ 3821.403995] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3824.129697] Key type lgssc unregistered [ 3824.479966] LNet: 109845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3824.490320] LNetError: 109845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.522575] LNet: Removed LNI 192.168.201.127@tcp [ 3825.404791] Key type .llcrypt unregistered [ 3825.407602] Key type ._llcrypt unregistered [ 3853.666387] Key type ._llcrypt registered [ 3853.671597] Key type .llcrypt registered [ 3853.862684] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3855.485132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3855.616816] alg: No test for adler32 (adler32-zlib) [ 3856.878634] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 3857.120245] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3858.799171] Key type lgssc registered [ 3860.411184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3871.691928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3871.717404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.599363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.837760] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3884.012241] LustreError: 111263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.040714] LustreError: 111263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3889.126541] LustreError: 111264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.058301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.569441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.789670] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3905.456452] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3914.754838] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3925.050502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3925.650480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.053834] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3944.245075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3944.276944] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3950.715219] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3955.609991] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3960.802269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.808941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.819825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.896211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.902203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.907973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.931323] Lustre: Skipped 1 previous similar message [ 3966.726885] Lustre: server umount lustre-OST0000 complete [ 3972.578050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3972.581271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.609088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.144172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.147651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.153496] Lustre: server umount lustre-MDT0000 complete [ 3980.262903] LustreError: 112853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.293180] LustreError: 112853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3982.775975] LustreError: 111242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552190 with bad export cookie 13748099133780584411 [ 3982.781340] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.789492] LustreError: 111242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3983.223533] Lustre: server umount lustre-MDT0001 complete [ 3997.824570] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4000.952755] Key type lgssc unregistered [ 4001.421472] LNet: 114908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4001.438309] LNetError: 114908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4001.455908] LNet: Removed LNI 192.168.201.127@tcp [ 4002.514633] Key type .llcrypt unregistered [ 4002.516199] Key type ._llcrypt unregistered [ 4022.491718] Key type ._llcrypt registered [ 4022.493717] Key type .llcrypt registered [ 4022.587638] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4024.016519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.101330] alg: No test for adler32 (adler32-zlib) [ 4025.399522] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 4025.621147] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4027.303769] Key type lgssc registered [ 4028.575564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.302804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4037.317108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.814205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.650577] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4049.382652] LustreError: 116129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.423505] LustreError: 116129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4052.930306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.428442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.400920] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4066.966647] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4072.792199] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4081.115885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.681316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.723096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4088.853416] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4098.205235] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4109.280885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.296326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.332594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.733417] Lustre: server umount lustre-OST0000 complete [ 4114.917062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4114.933876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.965417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4118.498201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.500860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4118.534902] Lustre: Skipped 1 previous similar message [ 4121.009468] Lustre: server umount lustre-MDT0000 complete [ 4123.642540] LustreError: 117124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.681887] LustreError: 117124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4126.209584] LustreError: 116110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552333 with bad export cookie 11800113156354479228 [ 4126.222654] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.228595] LustreError: 116110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4136.626368] Lustre: server umount lustre-MDT0001 complete [ 4149.434776] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4152.397921] Key type lgssc unregistered [ 4152.705319] LNet: 119668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4152.720269] LNetError: 119668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4153.767536] LNet: Removed LNI 192.168.201.127@tcp [ 4154.790929] Key type .llcrypt unregistered [ 4154.793396] Key type ._llcrypt unregistered [ 4183.956948] Key type ._llcrypt registered [ 4183.958856] Key type .llcrypt registered [ 4184.077509] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4185.896305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4186.012644] alg: No test for adler32 (adler32-zlib) [ 4187.240883] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 4187.615825] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4189.359362] Key type lgssc registered [ 4190.520315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.253395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4201.290481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.153636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.167724] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4212.712741] LustreError: 121088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.750869] LustreError: 121088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4217.832413] LustreError: 121089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.224029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.986925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.756438] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4238.355792] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4245.607488] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4254.626596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.052686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.045842] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4269.966882] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4273.494631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4282.338336] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4287.419463] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4287.424454] Lustre: Skipped 2 previous similar messages [ 4290.334148] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4290.396210] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4294.133148] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4294.144119] Lustre: Skipped 2 previous similar messages [ 4299.908790] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4299.911788] Lustre: Skipped 1 previous similar message [ 4305.774891] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4305.785699] Lustre: Skipped 2 previous similar messages [ 4315.727645] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.27@tcp 192.168.0.[2,10]@tcp [ 4321.351373] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.27/24@tcp in log lustre-MDT0000 [ 4321.358233] Lustre: Skipped 2 previous similar messages [ 4327.637948] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.27/24@tcp [ 4327.663951] Lustre: Skipped 1 previous similar message [ 4340.193629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.195241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.196189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4340.218040] Lustre: Skipped 1 previous similar message [ 4340.245852] Lustre: Skipped 1 previous similar message [ 4345.312829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4345.328443] Lustre: Skipped 1 previous similar message [ 4345.951937] Lustre: server umount lustre-OST0000 complete [ 4355.041608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4355.059176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.094156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.821857] Lustre: server umount lustre-MDT0000 complete [ 4360.672241] LustreError: 121083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.686543] LustreError: 121083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4360.847845] LustreError: 121067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552568 with bad export cookie 2489723435211949384 [ 4360.858183] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4360.861713] LustreError: 121067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4361.361609] Lustre: server umount lustre-MDT0001 complete [ 4373.505925] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4376.875933] Key type lgssc unregistered [ 4377.285479] LNet: 126138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4377.303997] LNetError: 126138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4377.344708] LNet: Removed LNI 192.168.201.127@tcp [ 4378.409160] Key type .llcrypt unregistered [ 4378.410975] Key type ._llcrypt unregistered [ 4410.534778] Key type ._llcrypt registered [ 4410.536790] Key type .llcrypt registered [ 4410.657618] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4412.027103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.053071] alg: No test for adler32 (adler32-zlib) [ 4413.154814] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 4413.499165] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4415.231152] Key type lgssc registered [ 4416.738560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4425.531832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4425.547542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.221214] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4427.268372] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.27/24@tcp [ 4427.294491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.815639] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4436.514711] LustreError: 127699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.524113] LustreError: 127699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4440.194437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.514213] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4440.542825] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.27/24@tcp [ 4440.617759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4444.933268] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4452.335376] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.360810] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4464.763963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.176522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.727522] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4472.386216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4478.909340] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4491.751100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.757400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.787069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4492.770536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4492.773125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.820867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4494.683516] Lustre: server umount lustre-OST0000 complete [ 4501.984405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4501.992319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.013955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.017824] Lustre: Skipped 1 previous similar message [ 4504.848809] Lustre: server umount lustre-MDT0000 complete [ 4508.129506] LustreError: 127696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.162597] LustreError: 127696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4508.991855] LustreError: 127682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552716 with bad export cookie 2109712376849354762 [ 4509.011944] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4509.591148] Lustre: server umount lustre-MDT0001 complete [ 4522.684416] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4526.124747] Key type lgssc unregistered [ 4526.622492] LNet: 131290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4526.631468] LNetError: 131290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.664695] LNet: Removed LNI 192.168.201.127@tcp [ 4527.951136] Key type .llcrypt unregistered [ 4527.957182] Key type ._llcrypt unregistered [ 4559.414892] Key type ._llcrypt registered [ 4559.416985] Key type .llcrypt registered [ 4559.549965] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4578.056205] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4579.804373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4579.979528] alg: No test for adler32 (adler32-zlib) [ 4581.151466] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 4581.558600] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4583.217042] Key type lgssc registered [ 4584.294631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4619.085547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4619.122450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.365369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4620.392264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4620.446170] Lustre: lustre-MDT0000: new disk, initializing [ 4620.495532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.510493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4625.364761] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4637.920581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.004930] Lustre: 134692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4638.059674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4638.063577] Lustre: Skipped 1 previous similar message [ 4638.159534] Lustre: lustre-MDT0001: new disk, initializing [ 4638.282512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4638.338847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4638.348690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4644.834284] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4657.237132] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4665.145287] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4674.168693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.412384] Lustre: lustre-OST0000: new disk, initializing [ 4674.415131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4674.420329] Lustre: 136664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4674.508434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.618208] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4684.391376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4684.405985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4684.476136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4694.035609] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4701.524283] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4701.792286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4708.327403] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4708.533731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4715.489686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4715.526612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.571746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4715.587481] Lustre: Skipped 1 previous similar message [ 4717.513177] Lustre: server umount lustre-OST0000 complete [ 4725.218823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4725.224755] LustreError: Skipped 1 previous similar message [ 4725.230841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.243521] Lustre: Skipped 1 previous similar message [ 4725.248757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.295896] Lustre: server umount lustre-MDT0000 complete [ 4734.061261] LustreError: 136663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552941 with bad export cookie 2186882244591720016 [ 4734.071671] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.075627] LustreError: 136663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4734.419108] Lustre: server umount lustre-MDT0001 complete [ 4744.994375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.577069] LustreError: 138881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.591557] LustreError: 138881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4745.649624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.718698] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4750.821797] LustreError: 138882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.938642] LustreError: 138881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.278162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.648130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4765.274303] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4774.997927] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4782.760830] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4790.986877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4791.279914] LustreError: 140702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.431130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.911148] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4809.539168] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4815.230562] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4815.491413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4820.821529] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4821.050588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4831.408536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.595236] Lustre: lustre-OST0001: new disk, initializing [ 4831.599246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4831.605761] Lustre: 142252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4831.709965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4838.787195] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4840.536547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4840.544502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4840.638920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4848.165297] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4848.377604] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4853.144285] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4853.354666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4860.416603] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90de05c06800.ost_server_uuid 50 [ 4862.212416] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90de05c06800.ost_server_uuid in IDLE state after 0 sec [ 4874.566814] Lustre: server umount lustre-OST0001 complete [ 4876.767925] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4876.778564] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.791939] Lustre: Skipped 1 previous similar message [ 4876.803448] LustreError: 142676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.818296] LustreError: 142676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4878.816663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.824328] LustreError: Skipped 1 previous similar message [ 4878.827525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.844635] Lustre: Skipped 1 previous similar message [ 4878.856219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4878.864400] Lustre: Skipped 1 previous similar message [ 4881.893119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4887.030825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4887.032065] LustreError: 142676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4887.039702] Lustre: Skipped 2 previous similar messages [ 4887.066916] LustreError: 142676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4893.151182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4893.384149] Lustre: server umount lustre-OST0000 complete [ 4897.263039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.280235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4897.287360] Lustre: Skipped 2 previous similar messages [ 4897.310980] Lustre: Skipped 3 previous similar messages [ 4903.733033] Lustre: server umount lustre-MDT0000 complete [ 4907.497516] LustreError: 139609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.516326] LustreError: 139609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4908.615806] LustreError: 138862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553116 with bad export cookie 2186882244591720905 [ 4908.616626] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4908.630198] LustreError: 138862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4909.437615] Lustre: server umount lustre-MDT0001 complete [ 4930.483727] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4933.596535] Key type lgssc unregistered [ 4933.982604] LNet: 145263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4934.013298] LNetError: 145263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4934.060612] LNet: Removed LNI 192.168.201.127@tcp [ 4935.308386] Key type .llcrypt unregistered [ 4935.312826] Key type ._llcrypt unregistered [ 4982.305032] Key type ._llcrypt registered [ 4982.310854] Key type .llcrypt registered [ 4982.427734] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4998.601118] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5000.149409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.229631] alg: No test for adler32 (adler32-zlib) [ 5001.264133] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5001.479335] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 5003.111587] Key type lgssc registered [ 5004.638485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5048.861258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5048.901925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.228651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5050.265296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5050.379133] Lustre: lustre-MDT0000: new disk, initializing [ 5050.545821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.577737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5057.343990] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5072.611348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.772748] Lustre: 149423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5072.932959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5072.943675] Lustre: Skipped 1 previous similar message [ 5073.100742] Lustre: lustre-MDT0001: new disk, initializing [ 5073.290738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5073.374111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5073.414298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5081.814059] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5092.898829] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5100.579947] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5111.606228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5112.031251] Lustre: lustre-OST0000: new disk, initializing [ 5112.043116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5112.054340] Lustre: 151393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5112.178315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.048376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5119.077260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5119.258913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.228971] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5132.967928] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5138.003880] Lustre: Failing over lustre-OST0000 [ 5138.028889] LustreError: 152200:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (00000000e9f3ecf7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5139.940405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.941840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5139.943756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.971239] Lustre: Skipped 1 previous similar message [ 5139.974372] LustreError: Skipped 1 previous similar message [ 5140.338809] Lustre: server umount lustre-OST0000 complete [ 5160.655363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.927871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.945950] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5166.519935] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5167.053427] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5170.358622] Lustre: Failing over lustre-MDT0000 [ 5170.602849] Lustre: server umount lustre-MDT0000 complete [ 5170.663631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5170.673485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.290401] LustreError: 149430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.305077] LustreError: 149430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5177.285169] LustreError: 150605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.386926] LustreError: 149434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.440640] LustreError: 149434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5182.417597] LustreError: 149431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.527094] Lustre: 147645:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553378/real 1787553378] req@000000002a3c8299 x1874385393729536/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787553394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5186.538386] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5186.542883] LustreError: 149435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.554050] LustreError: 149435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5190.349663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.770173] LustreError: 147644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009c00b921 x1874385393733760/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5196.788347] LustreError: 150605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.807223] LustreError: 150605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5197.045324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 5197.139051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.204240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5197.782806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5202.186753] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5202.420176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5202.465405] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5202.564678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5212.645439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.648860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.654570] Lustre: Skipped 3 previous similar messages [ 5212.682803] Lustre: Skipped 2 previous similar messages [ 5217.375657] Lustre: server umount lustre-OST0000 complete [ 5222.883206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.885141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5222.925251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.957305] Lustre: Skipped 1 previous similar message [ 5228.001252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5228.012300] Lustre: Skipped 1 previous similar message [ 5233.131975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.137417] Lustre: Skipped 1 previous similar message [ 5236.193185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5236.442862] Lustre: server umount lustre-MDT0000 complete [ 5238.240710] LustreError: 154094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.255599] LustreError: 154094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5240.073348] LustreError: 149414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553447 with bad export cookie 17439007563470215127 [ 5240.078033] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5240.081571] LustreError: 149414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.470695] Lustre: server umount lustre-MDT0001 complete [ 5250.346335] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 5254.371115] Key type lgssc unregistered [ 5254.961643] LNet: 155052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5254.968225] LNetError: 155052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5254.993744] LNet: Removed LNI 192.168.201.127@tcp [ 5256.529687] Key type .llcrypt unregistered [ 5256.532093] Key type ._llcrypt unregistered [ 5298.622891] Key type ._llcrypt registered [ 5298.627416] Key type .llcrypt registered [ 5298.762277] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5300.697396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5300.717107] alg: No test for adler32 (adler32-zlib) [ 5301.761841] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 5302.219282] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 5304.041060] Key type lgssc registered [ 5305.295356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5314.446749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5314.481291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.929968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.740061] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5326.309693] LustreError: 156577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5326.329678] LustreError: 156577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5330.686451] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5331.432870] LustreError: 156578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.418516] LustreError: 156606:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5335.433824] LustreError: 156606:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5335.447386] LustreError: 156606:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5335.856689] Lustre: server umount lustre-MDT0000 complete [ 5353.108753] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5365.357727] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5411.489660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.660355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.689550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.782118] Lustre: lustre-MDT0000: new disk, initializing [ 5411.923668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.948347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5417.653738] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5432.274043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.521036] Lustre: 159909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5432.602557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5432.606503] Lustre: Skipped 1 previous similar message [ 5432.710218] Lustre: lustre-MDT0001: new disk, initializing [ 5432.907956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5433.008361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5433.022444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5438.784958] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5450.090878] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5456.829392] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5465.233865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.560538] Lustre: lustre-OST0000: new disk, initializing [ 5465.566245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5465.578671] Lustre: 161880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5465.675495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.322555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5471.332376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5471.436128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5473.078084] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5483.358752] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5489.400771] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5492.129469] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:41:38 (1787553698) [ 5784.038485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.060756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5784.061954] Lustre: Skipped 1 previous similar message [ 5784.088699] Lustre: Skipped 2 previous similar messages [ 5788.626302] Lustre: server umount lustre-MDT0000 complete [ 5789.151502] LustreError: 160994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.176092] LustreError: 160994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5792.439512] LustreError: 159904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553999 with bad export cookie 18032534737333613324 [ 5792.440510] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5792.448076] LustreError: 159904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5794.271709] LustreError: 159917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.275977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.306823] Lustre: Skipped 1 previous similar message [ 5794.320360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5797.279794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5798.903656] Lustre: server umount lustre-MDT0001 complete [ 5809.645933] Lustre: server umount lustre-OST0000 complete [ 5820.861451] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5831.626584] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5873.400105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.751449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5873.776941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5873.882474] Lustre: lustre-MDT0000: new disk, initializing [ 5874.014900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.036425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.951564] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5895.987941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.222643] Lustre: 166563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5896.361143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5896.373641] Lustre: Skipped 1 previous similar message [ 5896.518565] Lustre: lustre-MDT0001: new disk, initializing [ 5896.724116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.805619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5896.829755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5903.751931] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5916.904840] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.674242] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5935.733444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.136469] Lustre: lustre-OST0000: new disk, initializing [ 5936.145920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5936.157790] Lustre: 168533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5936.258615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.854753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5942.881789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5943.056939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5943.365978] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5955.012279] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.264809] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5961.546584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5967.859749] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5968.052465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5973.986143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.986554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.010673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5974.025600] Lustre: Skipped 1 previous similar message [ 5976.750984] Lustre: server umount lustre-OST0000 complete [ 5983.715352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5983.723125] LustreError: Skipped 1 previous similar message [ 5983.725745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.737059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.743687] Lustre: Skipped 1 previous similar message [ 5989.271292] Lustre: server umount lustre-MDT0000 complete [ 5989.357349] LustreError: 166569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.378595] LustreError: 166569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5994.733989] LustreError: 166555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554202 with bad export cookie 18032534737333772161 [ 5994.773541] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5995.191894] Lustre: server umount lustre-MDT0001 complete [ 6007.060493] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:50:13 (1787554213) [ 6026.186660] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 6041.335357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6085.571904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.915721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6085.957924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6086.063333] Lustre: lustre-MDT0000: new disk, initializing [ 6086.169144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.186856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6091.679686] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6103.991161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.138098] Lustre: 173290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6104.515231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6109.959756] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6120.683453] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6127.435759] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6135.096711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6135.388440] Lustre: lustre-OST0000: new disk, initializing [ 6135.391456] Lustre: Skipped 1 previous similar message [ 6135.396106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6135.402207] Lustre: Skipped 2 previous similar messages [ 6135.410562] Lustre: 175269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6141.016074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6141.029780] Lustre: Skipped 1 previous similar message [ 6141.044717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6141.215986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6142.995700] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6155.269351] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6167.011450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.012654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6167.049849] Lustre: Skipped 1 previous similar message [ 6167.072075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6167.108187] Lustre: Skipped 3 previous similar messages [ 6171.780763] Lustre: server umount lustre-OST0000 complete [ 6177.255791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.035221] Lustre: server umount lustre-MDT0000 complete [ 6186.473846] LustreError: 173297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.486417] LustreError: 173297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6187.213298] LustreError: 173281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554394 with bad export cookie 18032534737333773043 [ 6187.242434] LustreError: 173281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6187.256735] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.518655] Lustre: server umount lustre-MDT0001 complete [ 6197.454186] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:53:24 (1787554404) [ 6213.113405] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 6222.739306] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6267.474504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.999413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6268.081980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6268.277093] Lustre: lustre-MDT0000: new disk, initializing [ 6268.474296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.479820] Lustre: Skipped 2 previous similar messages [ 6268.503873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6275.004069] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6286.256945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.529767] Lustre: 179597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6286.961885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6291.334594] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6300.501764] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6306.338525] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6313.748929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.002498] Lustre: 181574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6321.539542] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6322.755709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6322.941157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6332.801855] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6343.648976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6343.655041] LustreError: Skipped 1 previous similar message [ 6343.672129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.688965] Lustre: Skipped 1 previous similar message [ 6343.693395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6343.696829] Lustre: Skipped 5 previous similar messages [ 6348.007480] Lustre: server umount lustre-OST0000 complete [ 6353.393866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6353.404815] LustreError: Skipped 1 previous similar message [ 6358.507568] Lustre: server umount lustre-MDT0000 complete [ 6359.018418] LustreError: 179608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6359.039849] LustreError: 179608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6362.519319] LustreError: 179586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554570 with bad export cookie 18032534737333774128 [ 6362.530929] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6362.535657] LustreError: 179586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6362.800332] Lustre: server umount lustre-MDT0001 complete [ 6377.223889] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 6380.541390] Key type lgssc unregistered [ 6380.816028] LNet: 183433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6380.825846] LNetError: 183433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6381.864626] LNet: Removed LNI 192.168.201.127@tcp [ 6382.719218] Key type .llcrypt unregistered [ 6382.721759] Key type ._llcrypt unregistered [ 6407.052422] Key type ._llcrypt registered [ 6407.059195] Key type .llcrypt registered [ 6407.151867] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6408.453260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6408.493874] alg: No test for adler32 (adler32-zlib) [ 6409.422269] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6409.669299] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 6411.327500] Key type lgssc registered [ 6412.305641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6419.053544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6419.066787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.445563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6424.676428] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6429.667438] LustreError: 184853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.680184] LustreError: 184853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6432.003475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.435543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.684325] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6443.249206] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6449.219529] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6455.232542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6455.554427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.134167] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6464.905898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6470.829858] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6490.593133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.598634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.603347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.637311] Lustre: Skipped 1 previous similar message [ 6495.510835] Lustre: server umount lustre-OST0000 complete [ 6500.835930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.841352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.850240] Lustre: Skipped 2 previous similar messages [ 6505.965696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6505.978035] Lustre: Skipped 2 previous similar messages [ 6506.763467] Lustre: server umount lustre-MDT0000 complete [ 6511.072581] LustreError: 185905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.096969] LustreError: 185905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6511.155902] LustreError: 184833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554718 with bad export cookie 17566636171650357085 [ 6511.163359] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6511.171459] LustreError: 184833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6511.535526] Lustre: server umount lustre-MDT0001 complete [ 6523.437527] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 6526.717099] Key type lgssc unregistered [ 6527.050699] LNet: 188412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6527.060813] LNetError: 188412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6527.093540] LNet: Removed LNI 192.168.201.127@tcp [ 6528.251277] Key type .llcrypt unregistered [ 6528.259458] Key type ._llcrypt unregistered [ 6557.011291] Key type ._llcrypt registered [ 6557.015507] Key type .llcrypt registered [ 6557.151366] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6559.116851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6559.161344] alg: No test for adler32 (adler32-zlib) [ 6560.485862] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6560.794798] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 6562.623144] Key type lgssc registered [ 6564.881772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6576.074887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6576.091768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6577.575523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.762766] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6587.878255] LustreError: 189829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.897486] LustreError: 189829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6591.872771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.422928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6597.902436] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6606.324544] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.408474] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6621.086308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.402488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.494744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6628.991932] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6638.075122] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.243804] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 6647.274443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.281538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6647.298799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6647.311985] Lustre: Skipped 1 previous similar message [ 6651.352077] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 6651.622991] Lustre: server umount lustre-OST0000 complete [ 6661.586816] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90de04a79000.ost_server_uuid 50 [ 6663.351238] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6682.082824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.092726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.103071] Lustre: Skipped 2 previous similar messages [ 6686.058159] Lustre: server umount lustre-MDT0000 complete [ 6687.201049] LustreError: 190558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.233981] LustreError: 190558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6688.228684] LustreError: 189829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.255336] LustreError: 189829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6690.701457] LustreError: 189811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554898 with bad export cookie 13158411376726238427 [ 6690.703401] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.719245] LustreError: 189811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6691.185823] Lustre: server umount lustre-MDT0001 complete [ 6701.559575] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:01:48 (1787554908) [ 6708.908615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.370965] LustreError: 193757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.462428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.848361] LustreError: 193758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.392988] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6724.072489] LustreError: 193757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.090883] LustreError: 193757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6726.002541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.551247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.640986] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6743.436868] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6750.629979] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6758.359922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.714270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.439557] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6774.271239] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6779.193242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6781.890039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.132864] Lustre: lustre-OST0001: new disk, initializing [ 6782.136203] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6782.140277] Lustre: 196604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6782.212433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6788.405150] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6791.761852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6791.771912] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6791.829819] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6799.988048] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6804.961073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6804.969405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6804.984910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6804.999023] Lustre: Skipped 1 previous similar message [ 6807.009776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.017964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.212355] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 6816.228181] Lustre: Skipped 6 previous similar messages [ 6817.249120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6817.451746] Lustre: server umount lustre-OST0000 complete [ 6817.782040] LustreError: 195586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.793302] LustreError: 195586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6824.926747] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6825.129891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6830.079563] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6830.357408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6848.480448] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6848.496196] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6848.504156] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6848.511158] Lustre: Skipped 1 previous similar message [ 6848.532727] Lustre: Skipped 1 previous similar message [ 6851.158299] Lustre: server umount lustre-OST0001 complete [ 6858.720944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.750867] Lustre: Skipped 1 previous similar message [ 6862.685365] Lustre: server umount lustre-MDT0000 complete [ 6863.842799] LustreError: 194487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.880101] LustreError: 194487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 6867.932709] LustreError: 194496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555075 with bad export cookie 13158411376726239498 [ 6867.941923] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.951240] LustreError: 194496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6869.290891] Lustre: server umount lustre-MDT0001 complete [ 6903.545896] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:05:10 (1787555110) [ 6911.452237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.705172] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6911.736468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6911.865368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.075568] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6927.161829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.341358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6927.395216] Lustre: 200047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6933.021608] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6942.489633] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6949.781805] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.445925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.609556] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6966.100432] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6968.049320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6976.217924] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6984.274709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.363224] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6984.652116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6984.655053] Lustre: Skipped 2 previous similar messages [ 6991.741930] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7000.405142] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.693190] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 7006.697168] Lustre: Skipped 2 previous similar messages [ 7009.248498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7009.253036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.279357] LustreError: Skipped 2 previous similar messages [ 7010.466610] Lustre: server umount lustre-OST0000 complete [ 7011.779782] LustreError: 203413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7030.245727] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.763955] Lustre: server umount lustre-OST0001 complete [ 7046.979947] Lustre: server umount lustre-MDT0000 complete [ 7051.926109] LustreError: 201917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555259 with bad export cookie 13158411376726240905 [ 7051.929483] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7051.935431] LustreError: 201917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7052.837581] Lustre: server umount lustre-MDT0001 complete [ 7079.589653] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:08:06 (1787555286) [ 7098.712297] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 7110.921300] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7151.530163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.711696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7151.731107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7151.793812] Lustre: lustre-MDT0000: new disk, initializing [ 7151.832444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.872876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7156.222313] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7167.591591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.707715] Lustre: 208148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7167.758347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7167.762266] Lustre: Skipped 1 previous similar message [ 7167.827115] Lustre: lustre-MDT0001: new disk, initializing [ 7167.953390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7167.972546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7173.252435] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7185.009151] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.923813] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.401746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.811148] Lustre: lustre-OST0000: new disk, initializing [ 7199.817625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7199.828046] Lustre: 210119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7205.930228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7205.935156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7206.031820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7206.761654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7216.592266] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.196079] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7221.492970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7226.028321] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7226.291870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7231.462132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.482602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7231.506362] Lustre: Skipped 6 previous similar messages [ 7231.537213] Lustre: Skipped 7 previous similar messages [ 7234.675308] Lustre: server umount lustre-OST0000 complete [ 7239.652917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7239.661375] LustreError: Skipped 1 previous similar message [ 7245.341205] Lustre: server umount lustre-MDT0000 complete [ 7246.824322] LustreError: 208160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.879128] LustreError: 208160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7249.454432] LustreError: 208140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555456 with bad export cookie 13158411376726242466 [ 7249.459298] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.466506] LustreError: 208140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7249.712087] Lustre: server umount lustre-MDT0001 complete [ 7258.896363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.245848] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7275.948138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.033430] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7295.489529] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7302.061730] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.287143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.330502] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7327.182594] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7332.623106] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7332.821730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7337.368656] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7337.622266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7342.564222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7342.579927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.604948] Lustre: Skipped 2 previous similar messages [ 7354.335269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7354.507829] Lustre: server umount lustre-OST0000 complete [ 7361.156384] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7366.439697] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7396.218046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.679548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7399.708513] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7403.859096] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7405.863200] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7405.912165] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7412.853378] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7414.730701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7420.389347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7420.399176] LustreError: Skipped 1 previous similar message [ 7420.408719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7420.410885] Lustre: Skipped 8 previous similar messages [ 7426.559935] Lustre: server umount lustre-OST0000 complete [ 7437.940961] Lustre: server umount lustre-MDT0000 complete [ 7443.152184] LustreError: 212318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555650 with bad export cookie 13158411376726243348 [ 7443.157715] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7443.161240] LustreError: 212318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7443.715574] Lustre: server umount lustre-MDT0001 complete [ 7453.567958] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:14:20 (1787555660) [ 7460.683320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.015148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.022649] Lustre: Skipped 6 previous similar messages [ 7465.501587] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7475.511499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.161275] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7490.434804] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7497.383752] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7504.951678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.723374] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7521.196533] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7527.772323] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7528.006248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7532.997262] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7533.469318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7541.067782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.386384] Lustre: lustre-OST0001: new disk, initializing [ 7541.396556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7541.414817] Lustre: 221675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7549.092848] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7550.657363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7550.675728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7550.814051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7560.410840] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.719853] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7565.977681] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7571.780356] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7572.039984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7576.548507] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.555348] Lustre: Skipped 5 previous similar messages [ 7576.557968] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.577469] LustreError: Skipped 1 previous similar message [ 7588.322037] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7588.427571] Lustre: server umount lustre-OST0001 complete [ 7591.905991] LustreError: 220128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.952942] LustreError: 220128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7595.259901] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7595.537138] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7601.423591] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7601.632517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7632.272525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.496397] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7637.522417] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7637.529454] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7637.546732] Lustre: Skipped 1 previous similar message [ 7639.560490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7647.437924] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7649.110402] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7657.471454] Lustre: server umount lustre-OST0001 complete [ 7678.433939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7678.448616] Lustre: Skipped 18 previous similar messages [ 7681.176673] Lustre: server umount lustre-OST0000 complete [ 7691.674553] Lustre: server umount lustre-MDT0000 complete [ 7695.896238] LustreError: 218286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555903 with bad export cookie 13158411376726244314 [ 7695.903424] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.913640] LustreError: 218286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7702.534865] Lustre: server umount lustre-MDT0001 complete [ 7728.688646] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:18:55 (1787555935) [ 7735.331505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.679696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7735.726414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7740.660344] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7750.006585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.062745] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7750.081279] Lustre: 226922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7755.356115] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7763.980053] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7770.301369] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7777.249622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.548208] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7784.526609] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7792.900440] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.859698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7801.917287] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7808.583312] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7818.118988] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7819.924520] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7824.494977] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7824.655256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7829.350790] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7829.621223] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7836.119864] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90de09ea7800.ost_server_uuid 50 [ 7838.043699] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90de09ea7800.ost_server_uuid in IDLE state after 0 sec [ 7839.065413] Lustre: Permanently deactivating lustre-OST0001 [ 7839.075989] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7843.279368] Lustre: Permanently reactivating lustre-OST0001 [ 7843.314465] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7843.319196] Lustre: Skipped 2 previous similar messages [ 7848.421044] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7848.435139] LustreError: Skipped 5 previous similar messages [ 7848.444930] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.468097] Lustre: Skipped 7 previous similar messages [ 7853.369725] Lustre: server umount lustre-OST0001 complete [ 7863.339466] Lustre: server umount lustre-OST0000 complete [ 7881.183118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7881.399340] Lustre: server umount lustre-MDT0000 complete [ 7885.085355] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556092 with bad export cookie 13158411376726245602 [ 7885.096482] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7885.100209] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7885.534039] Lustre: server umount lustre-MDT0001 complete [ 7910.015655] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:21:57 (1787556117) [ 7917.683743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.892333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7917.916829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7917.924461] Lustre: Skipped 2 previous similar messages [ 7923.423930] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7932.671245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.756434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7932.778267] Lustre: 233620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7937.638666] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7946.418558] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7954.304701] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7964.291660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.556327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7964.629667] Lustre: Permanently deactivating lustre-OST0000 [ 7964.671426] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7970.859098] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7972.931131] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7981.453207] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7989.314298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7989.410711] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7989.664343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.669108] Lustre: Skipped 11 previous similar messages [ 7995.784948] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7999.729355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8004.838676] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8007.201449] Lustre: Permanently reactivating lustre-OST0000 [ 8007.217869] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8007.227641] Lustre: Skipped 2 previous similar messages [ 8008.195391] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8008.200554] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000bb207924, cur 1787556215 deadline 1787556208 last 1787556178 [ 8008.204280] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8008.229403] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8008.310403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8030.571729] Lustre: server umount lustre-OST0001 complete [ 8039.891364] Lustre: server umount lustre-OST0000 complete [ 8051.060839] Lustre: server umount lustre-MDT0000 complete [ 8054.578343] LustreError: 236506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556262 with bad export cookie 13158411376726247275 [ 8054.582638] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8054.596522] LustreError: 236506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8054.883853] Lustre: server umount lustre-MDT0001 complete [ 8066.831573] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8069.732698] Key type lgssc unregistered [ 8070.041146] LNet: 238508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8070.048684] LNetError: 238508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8070.063160] LNet: Removed LNI 192.168.201.127@tcp [ 8070.754968] Key type .llcrypt unregistered [ 8070.759875] Key type ._llcrypt unregistered [ 8096.535584] Key type ._llcrypt registered [ 8096.540729] Key type .llcrypt registered [ 8096.725444] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8098.234147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8098.271158] alg: No test for adler32 (adler32-zlib) [ 8099.429859] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8099.785082] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8101.559198] Key type lgssc registered [ 8102.471947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8111.768814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8111.787978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.314878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.941612] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8122.854078] LustreError: 239975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8122.873629] LustreError: 239975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8125.971062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.089955] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8126.112989] Lustre: 239963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8126.125581] Lustre: Permanently deactivating lustre-MDT0001 [ 8126.136941] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8126.466324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8131.423281] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8132.147450] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8139.365458] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8145.796574] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8152.172963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.420225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.553839] LustreError: 241818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8157.576279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8159.159659] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8162.794908] LustreError: 242108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8167.905151] LustreError: 241816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.678269] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.006678] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8176.080210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.108136] LustreError: 242108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8176.133188] LustreError: 242108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8176.256531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.757444] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8181.783767] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8184.251771] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8192.422324] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8196.444806] Lustre: Permanently reactivating lustre-MDT0001 [ 8196.465438] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8196.468475] Lustre: Skipped 2 previous similar messages [ 8199.865925] Lustre: lustre-MDT0001: Client e8862cf2-f84f-416a-aefa-ac57d0bdb2f3 (at 192.168.201.27@tcp) reconnecting [ 8204.378252] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.390352] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8204.398550] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8204.408586] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8205.843898] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8207.707704] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8213.859467] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8214.166501] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8219.512276] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8219.832150] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8222.937188] Lustre: Permanently deactivating lustre-MDT0001 [ 8222.988199] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8223.000884] Lustre: Skipped 2 previous similar messages [ 8231.057606] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8244.194266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8244.202793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.237813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.243352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.243631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.259739] Lustre: Skipped 1 previous similar message [ 8246.282140] Lustre: Skipped 1 previous similar message [ 8248.078835] Lustre: server umount lustre-MDT0000 complete [ 8251.360224] LustreError: 239976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.379021] LustreError: 239976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8251.895786] LustreError: 239955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556459 with bad export cookie 14542521056427066155 [ 8251.897535] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.907253] LustreError: 239955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8252.334938] Lustre: server umount lustre-MDT0001 complete [ 8258.823336] Lustre: server umount lustre-OST0000 complete [ 8271.895316] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:27:58 (1787556478) [ 8286.526437] Lustre: server umount lustre-OST0001 complete [ 8294.392880] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 8310.241711] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8352.845538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.235846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8353.256410] Lustre: Skipped 2 previous similar messages [ 8353.286296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8353.375844] Lustre: lustre-MDT0000: new disk, initializing [ 8353.471428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.506262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8359.103976] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8372.021214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.203950] Lustre: 248303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8372.316078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8372.328592] Lustre: Skipped 1 previous similar message [ 8372.465329] Lustre: lustre-MDT0001: new disk, initializing [ 8372.567469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8372.619182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8372.629387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8378.298707] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8390.157497] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.632147] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8404.442553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.771208] Lustre: lustre-OST0000: new disk, initializing [ 8404.778945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8404.791348] Lustre: 250273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8404.912657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.691391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8410.711340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8410.867272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8411.704827] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8422.007924] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8424.434260] LustreError: 248310:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8433.145840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.406235] Lustre: lustre-OST0001: new disk, initializing [ 8433.412552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8433.426941] Lustre: 251389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8434.458841] LustreError: 248310:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8434.506875] LustreError: 248312:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8440.093690] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8443.427944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8443.431753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8443.557727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8444.583184] LustreError: 248312:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8449.961845] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8451.863487] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8454.114790] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.121830] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.149714] Lustre: Skipped 1 previous similar message [ 8454.166106] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8454.172043] Lustre: Skipped 2 previous similar messages [ 8458.720723] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8458.725814] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8458.728171] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.753143] Lustre: Skipped 1 previous similar message [ 8459.888048] Lustre: server umount lustre-OST0001 complete [ 8463.849756] LustreError: 250259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.867238] LustreError: 250259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8477.668180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8477.670825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.676499] LustreError: Skipped 1 previous similar message [ 8477.678963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.695521] Lustre: Skipped 1 previous similar message [ 8477.716981] Lustre: Skipped 1 previous similar message [ 8482.901631] Lustre: server umount lustre-OST0000 complete [ 8489.441511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.445162] Lustre: Skipped 3 previous similar messages [ 8494.257437] Lustre: server umount lustre-MDT0000 complete [ 8499.428714] LustreError: 248297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556706 with bad export cookie 14542521056427068969 [ 8499.433094] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8499.441601] LustreError: 248297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8506.032234] Lustre: server umount lustre-MDT0001 complete [ 8521.037735] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8524.657046] Key type lgssc unregistered [ 8524.848952] LNet: 253426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8524.853518] LNetError: 253426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.869102] LNet: Removed LNI 192.168.201.127@tcp [ 8525.980917] Key type .llcrypt unregistered [ 8525.982911] Key type ._llcrypt unregistered [ 8569.105797] Key type ._llcrypt registered [ 8569.108266] Key type .llcrypt registered [ 8569.215654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8570.737465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8570.764594] alg: No test for adler32 (adler32-zlib) [ 8571.996428] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8572.466696] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8574.295915] Key type lgssc registered [ 8575.484363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8585.418848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8585.478782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8586.859214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8586.887393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8587.084628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.481980] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8600.409865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.603331] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8600.665921] Lustre: 255636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8600.909020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.358770] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8614.331866] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.206830] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8629.799273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8630.076242] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8630.404991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.126763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8637.885354] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8647.606140] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8702.948812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8702.950195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.987285] LustreError: Skipped 1 previous similar message [ 8702.989151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8703.066230] Lustre: Skipped 1 previous similar message [ 8703.860769] Lustre: server umount lustre-OST0000 complete [ 8709.952218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8723.123767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.623242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.075399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8729.103741] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8729.104643] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8731.115354] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8739.335363] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8780.258424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8780.262071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.266711] LustreError: Skipped 1 previous similar message [ 8780.270159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8780.276658] Lustre: Skipped 1 previous similar message [ 8780.299757] Lustre: Skipped 1 previous similar message [ 8784.693790] Lustre: server umount lustre-OST0000 complete [ 8790.496068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8790.502497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.504881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8790.516355] Lustre: Skipped 1 previous similar message [ 8790.529254] Lustre: Skipped 1 previous similar message [ 8795.065232] Lustre: server umount lustre-MDT0000 complete [ 8795.616235] LustreError: 255647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.633026] LustreError: 255647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8798.744408] LustreError: 255629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557006 with bad export cookie 18147677166132773659 [ 8798.750745] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.753451] LustreError: 255629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8799.194078] Lustre: server umount lustre-MDT0001 complete [ 8811.561378] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8814.502605] Key type lgssc unregistered [ 8814.825248] LNet: 261062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8814.833965] LNetError: 261062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8815.851991] LNet: Removed LNI 192.168.201.127@tcp [ 8816.739835] Key type .llcrypt unregistered [ 8816.742063] Key type ._llcrypt unregistered [ 8843.671064] Key type ._llcrypt registered [ 8843.672993] Key type .llcrypt registered [ 8843.776802] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8844.869683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8844.912624] alg: No test for adler32 (adler32-zlib) [ 8846.295367] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8846.529738] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8848.223846] Key type lgssc registered [ 8849.580165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8859.238475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8859.262057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.959562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.924725] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8871.399399] LustreError: 262480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.446159] LustreError: 262480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8876.520116] LustreError: 262499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8879.034203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.518177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.570825] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8895.995518] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8902.979159] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8911.601347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8911.932513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.262746] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8927.993306] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8931.467641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8960.992537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8961.009780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8961.032308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8962.547376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8962.548770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.551091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8964.418096] Lustre: server umount lustre-OST0000 complete [ 8971.745596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8971.752245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8971.765162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8971.770168] Lustre: Skipped 1 previous similar message [ 8974.728448] Lustre: server umount lustre-MDT0000 complete [ 8977.895456] LustreError: 262480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.926268] LustreError: 262480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8978.597289] LustreError: 262462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557186 with bad export cookie 4063101860207564987 [ 8978.600696] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8978.622156] LustreError: 262462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8979.050899] Lustre: server umount lustre-MDT0001 complete [ 8992.873409] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8997.682433] Key type lgssc unregistered [ 8998.260053] LNet: 266791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8998.262965] LNetError: 266791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8999.335055] LNet: Removed LNI 192.168.201.127@tcp [ 9000.885520] Key type .llcrypt unregistered [ 9000.887712] Key type ._llcrypt unregistered [ 9021.486167] Key type ._llcrypt registered [ 9021.488304] Key type .llcrypt registered [ 9021.627620] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9023.134679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9023.280125] alg: No test for adler32 (adler32-zlib) [ 9024.397762] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9024.599954] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9026.239450] Key type lgssc registered [ 9027.406910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9035.036306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9035.072574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.440707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9041.556821] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9045.986046] LustreError: 268014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9046.007760] LustreError: 268014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9050.818805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.109144] LustreError: 268015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.529523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9056.764464] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9066.330415] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9073.976716] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9081.142219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9081.321302] LustreError: 269836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9081.534899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9086.990498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9088.423572] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9096.966475] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9111.520569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.524906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9111.543945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9116.611097] Lustre: server umount lustre-OST0000 complete [ 9121.764437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.773796] Lustre: Skipped 2 previous similar messages [ 9121.777155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9121.791035] Lustre: Skipped 2 previous similar messages [ 9126.885898] LustreError: 268015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.942702] Lustre: server umount lustre-MDT0000 complete [ 9130.372503] LustreError: 267995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557337 with bad export cookie 16792577294130376462 [ 9130.382434] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9130.385074] LustreError: 267995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9130.834288] Lustre: server umount lustre-MDT0001 complete [ 9142.801151] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9145.226692] Key type lgssc unregistered [ 9145.477481] LNet: 271696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9145.497226] LNetError: 271696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9145.516894] LNet: Removed LNI 192.168.201.127@tcp [ 9146.437228] Key type .llcrypt unregistered [ 9146.440877] Key type ._llcrypt unregistered [ 9179.732678] Key type ._llcrypt registered [ 9179.733845] Key type .llcrypt registered [ 9179.817263] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9181.780352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9181.867318] alg: No test for adler32 (adler32-zlib) [ 9183.010788] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9183.222782] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9184.935199] Key type lgssc registered [ 9186.818795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9197.375249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9197.392330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.902417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.133340] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9209.324446] LustreError: 273114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.348985] LustreError: 273114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9213.114328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.607919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9218.326680] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9227.651211] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9235.261475] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9245.291319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9246.136289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.087303] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9265.161848] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9268.768041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9304.545251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.558353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.567568] Lustre: Skipped 1 previous similar message [ 9304.581377] Lustre: Skipped 1 previous similar message [ 9307.311098] Lustre: server umount lustre-OST0000 complete [ 9314.789619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.807230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.810613] Lustre: Skipped 1 previous similar message [ 9314.823737] Lustre: Skipped 1 previous similar message [ 9318.464352] Lustre: server umount lustre-MDT0000 complete [ 9319.905465] LustreError: 273841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.926608] LustreError: 273841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9322.364323] LustreError: 273093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557529 with bad export cookie 1868561604974980254 [ 9322.365536] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.370968] LustreError: 273093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9322.717298] Lustre: server umount lustre-MDT0001 complete [ 9336.811988] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9340.409680] Key type lgssc unregistered [ 9340.671817] LNet: 277426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9340.679607] LNetError: 277426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9341.766822] LNet: Removed LNI 192.168.201.127@tcp [ 9342.760103] Key type .llcrypt unregistered [ 9342.763761] Key type ._llcrypt unregistered [ 9366.061613] Key type ._llcrypt registered [ 9366.064343] Key type .llcrypt registered [ 9366.183075] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9367.414163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9367.633135] alg: No test for adler32 (adler32-zlib) [ 9368.660906] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9368.959528] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9370.743203] Key type lgssc registered [ 9372.274480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9380.235041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9380.297368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.860909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.610310] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9392.097850] LustreError: 278669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9392.124721] LustreError: 278669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9396.492675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.815946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9402.005440] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9411.569849] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9417.997532] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9425.210263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9425.548537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9426.623990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9432.916781] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9441.581169] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9457.641475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9457.652458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9457.657977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9457.664979] LustreError: Skipped 1 previous similar message [ 9457.671419] Lustre: Skipped 1 previous similar message [ 9461.534473] Lustre: server umount lustre-OST0000 complete [ 9467.878071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.898229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9467.900816] Lustre: Skipped 2 previous similar messages [ 9467.917664] Lustre: Skipped 1 previous similar message [ 9472.855742] Lustre: server umount lustre-MDT0000 complete [ 9472.999689] LustreError: 278649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9473.048308] LustreError: 278649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9477.491212] LustreError: 278634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557685 with bad export cookie 3051225012896091732 [ 9477.493146] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9477.504800] LustreError: 278634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9478.063398] Lustre: server umount lustre-MDT0001 complete [ 9492.088984] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9494.590114] Key type lgssc unregistered [ 9494.968875] LNet: 282375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9494.997153] LNetError: 282375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9495.014202] LNet: Removed LNI 192.168.201.127@tcp [ 9496.075313] Key type .llcrypt unregistered [ 9496.077079] Key type ._llcrypt unregistered [ 9510.140912] Key type ._llcrypt registered [ 9510.143684] Key type .llcrypt registered [ 9510.254335] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9532.489788] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9554.593306] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9556.724516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9556.948676] alg: No test for adler32 (adler32-zlib) [ 9558.546636] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9558.991209] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9560.759252] Key type lgssc registered [ 9562.794783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9602.294934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9602.379441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.768813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9603.819913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9603.925258] Lustre: lustre-MDT0000: new disk, initializing [ 9604.072587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9604.090971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9608.789900] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9621.614506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.798841] Lustre: 285936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9621.861286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9621.869103] Lustre: Skipped 1 previous similar message [ 9621.994951] Lustre: lustre-MDT0001: new disk, initializing [ 9622.165888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9622.260233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9622.270510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9627.342812] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9638.646961] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9645.784254] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9654.725199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9655.051593] Lustre: lustre-OST0000: new disk, initializing [ 9655.054565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9655.059600] Lustre: 287907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9655.155640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.329289] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9663.605992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9663.638361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9663.832061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9675.071390] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9681.320663] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9681.509560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9686.675045] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9686.915660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9689.577110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9689.585357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9689.615083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9694.697131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9694.714540] Lustre: Skipped 1 previous similar message [ 9699.835953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9699.845753] Lustre: Skipped 2 previous similar messages [ 9703.391173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9703.794754] Lustre: server umount lustre-OST0000 complete [ 9709.024551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9709.030136] LustreError: Skipped 1 previous similar message [ 9709.035148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.067976] Lustre: Skipped 1 previous similar message [ 9709.074463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9710.049867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.228618] Lustre: server umount lustre-MDT0000 complete [ 9720.297167] LustreError: 285948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.339395] LustreError: 285948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9720.477099] LustreError: 285930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557928 with bad export cookie 18144340349491309236 [ 9720.490425] LustreError: 285930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9720.497901] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9720.940561] Lustre: server umount lustre-MDT0001 complete [ 9732.650621] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:52:19 (1787557939) [ 9741.150634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.935452] LustreError: 290324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.076976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.433165] LustreError: 290325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.461973] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9747.466021] LustreError: 290325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9751.522973] LustreError: 290324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.666423] LustreError: 290324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.025333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.313741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.381765] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9770.070770] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9777.256927] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.276728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.668180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.023750] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9798.460028] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9812.448207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9812.449677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.460714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9812.464578] Lustre: Skipped 1 previous similar message [ 9812.474570] Lustre: Skipped 3 previous similar messages [ 9819.228391] Lustre: server umount lustre-OST0000 complete [ 9827.810124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9827.846210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.855752] Lustre: Skipped 3 previous similar messages [ 9833.544356] Lustre: server umount lustre-MDT0000 complete [ 9838.048917] LustreError: 290325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9838.065915] LustreError: 290325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9838.309823] LustreError: 290306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558045 with bad export cookie 18144340349491310118 [ 9838.317042] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.320994] LustreError: 290306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9838.605068] Lustre: server umount lustre-MDT0001 complete [ 9853.306927] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9856.991972] Key type lgssc unregistered [ 9857.273384] LNet: 293885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9857.293321] LNetError: 293885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9857.331257] LNet: Removed LNI 192.168.201.127@tcp [ 9858.588783] Key type .llcrypt unregistered [ 9858.589978] Key type ._llcrypt unregistered [ 9895.439915] Key type ._llcrypt registered [ 9895.446497] Key type .llcrypt registered [ 9895.616178] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9918.242347] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9919.499557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9919.654612] alg: No test for adler32 (adler32-zlib) [ 9921.212422] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9921.417682] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9923.071185] Key type lgssc registered [ 9924.389581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9979.486064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9979.519381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.767048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9980.844076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9981.009600] Lustre: lustre-MDT0000: new disk, initializing [ 9981.176827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9981.240235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9986.850423] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9997.606647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.726036] Lustre: 297732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9997.755686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9997.758419] Lustre: Skipped 1 previous similar message [ 9997.817446] Lustre: lustre-MDT0001: new disk, initializing [ 9997.910401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9997.957965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9997.978438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10003.362346] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10013.572156] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10019.993435] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10027.850029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.022641] Lustre: lustre-OST2710: new disk, initializing [10028.026118] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [10028.031404] Lustre: 299700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10028.098859] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10035.157810] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10035.800476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10035.813695] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10035.959768] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [10049.045755] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10057.353819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.510461] Lustre: lustre-OST03e8: new disk, initializing [10057.514169] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10057.521912] Lustre: 300763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10057.608272] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10064.060246] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10065.522229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10065.540737] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10065.635438] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10072.926374] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10080.547377] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10080.750899] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10085.425662] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10085.614691] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10091.655114] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10091.901771] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10097.383680] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10097.586603] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10104.459567] Lustre: server umount lustre-MDT0000 complete [10105.315329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10105.322240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.857152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.869057] Lustre: Skipped 1 previous similar message [10108.337382] LustreError: 297725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558315 with bad export cookie 2842659898546009445 [10108.342057] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.346674] LustreError: 297725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10108.775969] Lustre: server umount lustre-MDT0001 complete [10123.954198] Lustre: server umount lustre-OST2710 complete [10138.397622] Lustre: server umount lustre-OST03e8 complete [10158.833390] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [10171.715964] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10214.857176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.197884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10215.272943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10215.377255] Lustre: lustre-MDT0000: new disk, initializing [10215.468450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.495152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10221.475337] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10233.478452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.598669] Lustre: 306016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10233.648558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10233.652402] Lustre: Skipped 1 previous similar message [10233.726961] Lustre: lustre-MDT0001: new disk, initializing [10233.889973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10233.937725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10233.957120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10238.762303] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10248.946764] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10257.027134] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10266.131382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.671779] Lustre: lustre-OST0000: new disk, initializing [10266.676101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10266.680502] Lustre: 308000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10266.760436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10274.876273] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10274.956563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10274.969359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10275.223587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10286.819304] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10292.097364] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10292.317672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10297.440575] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10297.705978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10300.904547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10300.923414] Lustre: Skipped 1 previous similar message [10300.933629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10300.944270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10300.948065] Lustre: Skipped 1 previous similar message [10306.017439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.029487] Lustre: Skipped 1 previous similar message [10311.148322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10311.154570] Lustre: Skipped 1 previous similar message [10314.207617] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10314.531482] Lustre: server umount lustre-OST0000 complete [10320.867381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10320.875231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.890722] Lustre: Skipped 1 previous similar message [10320.896378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10324.684778] Lustre: server umount lustre-MDT0000 complete [10326.502698] LustreError: 306024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.532436] LustreError: 306024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10329.299514] LustreError: 306010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558536 with bad export cookie 2842659898546011132 [10329.312400] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.317373] LustreError: 306010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10329.892448] Lustre: server umount lustre-MDT0001 complete [10341.994610] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:02:27 (1787558547) [10343.985067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10346.711141] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:02:33 (1787558553) [10351.698909] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10383.479111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.730583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10383.747915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10383.931699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.445524] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10398.098948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.218501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10398.250868] Lustre: 311861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10402.841968] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10425.848406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10426.069668] Lustre: Denying initial registration attempt from nid 192.168.201.127@tcp, specified as failover [10426.076361] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10426.081135] LustreError: 313313:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [10426.085745] LustreError: 313313:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10426.090085] LustreError: 313313:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10426.139221] Lustre: server umount lustre-OST0000 complete [10426.141132] LustreError: 313313:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10434.530802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10434.538639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10434.557899] Lustre: Skipped 1 previous similar message [10434.566949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10434.570201] Lustre: Skipped 2 previous similar messages [10438.521087] Lustre: server umount lustre-MDT0000 complete [10440.677918] LustreError: 311871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.707115] LustreError: 311871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10442.097726] LustreError: 311853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558649 with bad export cookie 2842659898546012014 [10442.101906] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10442.103336] LustreError: 311853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10442.472298] Lustre: server umount lustre-MDT0001 complete [10454.626395] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10457.717650] Key type lgssc unregistered [10457.999598] LNet: 314435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10458.011375] LNetError: 314435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10458.026164] LNet: Removed LNI 192.168.201.127@tcp [10458.880847] Key type .llcrypt unregistered [10458.884548] Key type ._llcrypt unregistered [10473.097413] Key type ._llcrypt registered [10473.099517] Key type .llcrypt registered [10473.297046] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10474.616067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10474.674450] alg: No test for adler32 (adler32-zlib) [10476.057966] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10476.492325] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10478.295290] Key type lgssc registered [10480.193865] Lustre: Echo OBD driver; http://www.lustre.org/ [10540.303365] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10550.341556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10550.361855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.662794] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10551.707863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10551.974314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10557.391975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10566.934291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10566.988094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10567.010327] Lustre: 317206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10567.277372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10572.404664] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10581.764775] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10588.408868] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10609.147813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10609.327396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10609.573469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10618.009030] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10630.104175] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10634.211169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10634.227501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.238035] LustreError: Skipped 1 previous similar message [10634.240497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10634.273532] Lustre: Skipped 1 previous similar message [10638.940388] Lustre: server umount lustre-OST0000 complete [10643.936451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.945506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.961386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10643.964083] Lustre: Skipped 1 previous similar message [10649.571703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10649.584336] Lustre: Skipped 3 previous similar messages [10649.783875] Lustre: server umount lustre-MDT0000 complete [10653.629117] LustreError: 317956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558861 with bad export cookie 17342181928967519666 [10653.632495] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10653.638845] LustreError: 317956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10653.978686] Lustre: server umount lustre-MDT0001 complete [10665.798110] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10669.102990] Key type lgssc unregistered [10669.521478] LNet: 320785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10669.528616] LNetError: 320785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10669.553117] LNet: Removed LNI 192.168.201.127@tcp [10670.315789] Key type .llcrypt unregistered [10670.317691] Key type ._llcrypt unregistered [10698.900759] Key type ._llcrypt registered [10698.902801] Key type .llcrypt registered [10699.114089] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10700.633630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10700.816715] alg: No test for adler32 (adler32-zlib) [10701.951266] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10702.285109] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10704.030843] Key type lgssc registered [10705.724895] Lustre: Echo OBD driver; http://www.lustre.org/ [10714.109869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10714.146322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.606737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.420081] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10725.856707] LustreError: 322203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.883424] LustreError: 322203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10728.909264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.250640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10734.806601] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10743.422260] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10749.615418] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10758.845989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.408479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.268739] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10775.034651] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10784.237250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.240951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10784.250416] Lustre: Skipped 2 previous similar messages [10784.259648] Lustre: Skipped 2 previous similar messages [10786.771673] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [10788.748818] Lustre: server umount lustre-MDT0000 complete [10789.355698] LustreError: 324696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.380487] LustreError: 324696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10791.876575] LustreError: 322198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.331652] LustreError: 322185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559000 with bad export cookie 9886430977850658490 [10793.368932] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10793.378524] LustreError: 322185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10794.466671] LustreError: 322200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10794.469724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.523710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10797.045148] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [10798.496333] LustreError: 322199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.530049] LustreError: 322199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10799.836776] Lustre: server umount lustre-MDT0001 complete [10807.726953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10817.713823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.015928] LustreError: 321511:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000140f0afe x1874391370604416/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10818.352911] LustreError: 324034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.431667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10818.486331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10823.597384] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10827.627691] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10832.376160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.790636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10837.211816] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10837.997455] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10845.089689] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10846.847732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10852.553322] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10854.141779] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10859.491545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10859.496362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.507567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10859.520244] Lustre: Skipped 1 previous similar message [10863.586260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10863.610824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10865.788850] Lustre: server umount lustre-OST0000 complete [10873.824433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10873.830299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.838399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.861984] Lustre: Skipped 1 previous similar message [10873.883433] Lustre: Skipped 3 previous similar messages [10884.065809] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10884.346962] Lustre: server umount lustre-MDT0000 complete [10888.866023] LustreError: 322185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559096 with bad export cookie 9886430977850666057 [10888.871181] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.903476] LustreError: 322185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10889.703880] Lustre: server umount lustre-MDT0001 complete [10901.337973] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10904.480685] Key type lgssc unregistered [10904.747580] LNet: 328479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10904.753320] LNetError: 328479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10905.766267] LNet: Removed LNI 192.168.201.127@tcp [10906.839769] Key type .llcrypt unregistered [10906.841455] Key type ._llcrypt unregistered [10931.283875] Key type ._llcrypt registered [10931.285675] Key type .llcrypt registered [10931.366969] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10932.201970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10932.342839] alg: No test for adler32 (adler32-zlib) [10933.484202] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [10933.861780] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10935.626859] Key type lgssc registered [10937.701725] Lustre: Echo OBD driver; http://www.lustre.org/ [10945.561507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10945.576444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10946.891074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10951.456708] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10957.283373] LustreError: 329894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.312376] LustreError: 329894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10960.169788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10960.701396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10966.480313] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10974.910984] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10982.680285] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10986.469073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10986.475551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10986.484217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10987.490341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10987.492912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.610025] LustreError: 329895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.627878] LustreError: 329895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10992.725722] Lustre: server umount lustre-MDT0000 complete [10997.445225] LustreError: 329876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559204 with bad export cookie 15040947105047952147 [10997.457712] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10997.465399] LustreError: 329876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10998.092061] Lustre: server umount lustre-MDT0001 complete [11013.026851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11013.476433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11013.526268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11013.717364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11018.477718] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11027.406061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.526343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11027.585897] Lustre: 332483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11027.816310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11031.960770] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11039.619261] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11046.536142] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11055.995225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11056.321656] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11056.334250] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11056.345423] LustreError: 334366:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [11056.356659] LustreError: 334366:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [11056.364403] LustreError: 334366:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11056.419115] Lustre: server umount lustre-OST0000 complete [11056.421958] LustreError: 334366:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11066.409219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11066.573584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11066.742725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11072.670524] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11073.714832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11080.279946] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11088.756920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11088.883670] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11088.991727] Lustre: lustre-OST0001: new disk, initializing [11088.995659] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11089.000886] Lustre: 335907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11089.097709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11096.072956] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11096.217693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11096.230519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11096.364599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11105.037745] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11111.904521] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11111.924198] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.932746] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11111.936249] Lustre: Skipped 1 previous similar message [11111.957339] Lustre: Skipped 2 previous similar messages [11114.737591] Lustre: server umount lustre-OST0001 complete [11117.027856] LustreError: 334832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.050690] LustreError: 334832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11119.584206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11119.588035] LustreError: Skipped 1 previous similar message [11119.589837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.596588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11122.153676] LustreError: 334832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.173102] LustreError: 334832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11125.040203] Lustre: server umount lustre-OST0000 complete [11130.336622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11130.350375] LustreError: Skipped 1 previous similar message [11130.353151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.364810] Lustre: Skipped 1 previous similar message [11130.368822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11130.371520] Lustre: Skipped 3 previous similar messages [11136.366962] Lustre: server umount lustre-MDT0000 complete [11137.518022] LustreError: 332494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11137.533866] LustreError: 332494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11140.983929] LustreError: 332476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559348 with bad export cookie 15040947105047952700 [11140.991187] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11141.003302] LustreError: 332476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11141.307194] Lustre: server umount lustre-MDT0001 complete [11153.163173] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [11155.882580] Key type lgssc unregistered [11156.249642] LNet: 337871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11156.262839] LNetError: 337871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11156.279834] LNet: Removed LNI 192.168.201.127@tcp [11157.106667] Key type .llcrypt unregistered [11157.108263] Key type ._llcrypt unregistered [11234.277147] Key type ._llcrypt registered [11234.279274] Key type .llcrypt registered [11234.389924] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [11251.298507] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11252.354632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11252.366041] alg: No test for adler32 (adler32-zlib) [11253.441847] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11253.710385] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [11255.370194] Key type lgssc registered [11256.543832] Lustre: Echo OBD driver; http://www.lustre.org/ [11291.715593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11291.732537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.105132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11293.170818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11293.319771] Lustre: lustre-MDT0000: new disk, initializing [11293.464442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11293.484195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11299.144720] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11313.917958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.145150] Lustre: 342812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11314.275156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11314.282206] Lustre: Skipped 1 previous similar message [11314.410316] Lustre: lustre-MDT0001: new disk, initializing [11314.557299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11314.605482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11314.640803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11320.058342] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11332.244173] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11339.533868] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11347.657560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.087916] Lustre: lustre-OST0000: new disk, initializing [11348.094042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11348.108213] Lustre: 344779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.255483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11354.020551] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11356.754648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11356.770073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11356.930497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11363.803659] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11368.995246] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11369.193368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11373.409878] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11373.607676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11377.634310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11377.637190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11377.653962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11377.670624] Lustre: Skipped 1 previous similar message [11381.476919] Lustre: server umount lustre-OST0000 complete [11386.336805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11386.345619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11386.371120] Lustre: Skipped 1 previous similar message [11386.382730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.873824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.876832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.963150] Lustre: server umount lustre-MDT0000 complete [11395.875816] LustreError: 342804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559603 with bad export cookie 3280983083960158515 [11395.880255] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11395.888794] LustreError: 342804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11396.250390] Lustre: server umount lustre-MDT0001 complete [11409.243937] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:20:15 (1787559615) [11420.726349] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:20:27 (1787559627) [11430.117994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.657252] LustreError: 347535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.717816] LustreError: 347535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11430.866529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.759464] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11436.010280] LustreError: 347536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11439.077203] LustreError: 347535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.126779] LustreError: 347536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11445.875918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.351638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11452.520441] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11462.215734] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.675402] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11477.301254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11477.753100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11486.309931] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11495.659546] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11509.332577] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11511.609671] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11514.188757] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11516.033946] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11518.301181] Lustre: Failing over lustre-MDT0000 [11518.685628] Lustre: server umount lustre-MDT0000 complete [11522.021987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.028425] Lustre: Skipped 1 previous similar message [11522.029974] LustreError: 347535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.052668] LustreError: 347535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11532.260963] LustreError: 348461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.311450] LustreError: 348461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11538.399192] Lustre: 341034:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787559729/real 1787559729] req@0000000096b2c171 x1874391949453184/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787559745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11538.427196] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.724790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.623893] LustreError: 347530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.654285] LustreError: 341031:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009ed95503 x1874391949458944/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11548.662910] LustreError: 347530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11548.943756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11548.996523] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11552.905937] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11553.732546] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11554.310245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11554.402907] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11554.511443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11563.588369] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11565.302452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11569.562537] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11572.069108] Lustre: Failing over lustre-MDT0000 [11572.587505] Lustre: server umount lustre-MDT0000 complete [11574.763663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.781132] Lustre: Skipped 3 previous similar messages [11584.453225] LustreError: 347531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.479779] LustreError: 347531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11591.136851] Lustre: 341034:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787559782/real 1787559782] req@0000000080851439 x1874391949474432/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787559798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11591.161489] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11591.697231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.379611] LustreError: 341031:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1469b11 x1874391949479936/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11601.698447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11601.774864] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11604.940417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11606.740306] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11607.031297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11607.040280] Lustre: Skipped 2 previous similar messages [11607.071069] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11607.115406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11616.948612] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11618.885935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11627.490773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11627.495963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11627.498281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11627.511074] Lustre: Skipped 1 previous similar message [11627.528253] Lustre: Skipped 3 previous similar messages [11631.742218] Lustre: server umount lustre-MDT0000 complete [11635.430665] LustreError: 347517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559842 with bad export cookie 3280983083960161231 [11635.433330] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11635.441942] LustreError: 347517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11635.794774] Lustre: server umount lustre-MDT0001 complete [11649.901104] Lustre: server umount lustre-OST0000 complete [11672.796227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11674.453567] LustreError: 353904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.499689] LustreError: 353904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [11674.690827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11680.994293] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11693.899962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.079243] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11710.596616] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11718.703190] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11726.368887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11734.257892] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11742.933328] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11745.272506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11759.584879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11759.591660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.598268] Lustre: Skipped 1 previous similar message [11759.601129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11762.281026] Lustre: server umount lustre-OST0000 complete [11770.852239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11770.861299] Lustre: Skipped 2 previous similar messages [11774.005316] Lustre: server umount lustre-MDT0000 complete [11778.534326] LustreError: 353886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559986 with bad export cookie 3280983083960161973 [11778.536759] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11778.543116] LustreError: 353886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11778.838399] Lustre: server umount lustre-MDT0001 complete [11793.541448] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [11796.666483] Key type lgssc unregistered [11797.053716] LNet: 357465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11797.062630] LNetError: 357465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11797.091459] LNet: Removed LNI 192.168.201.127@tcp [11797.839545] Key type .llcrypt unregistered [11797.841362] Key type ._llcrypt unregistered [11826.627599] Key type ._llcrypt registered [11826.630889] Key type .llcrypt registered [11826.726636] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11827.628184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11827.675947] alg: No test for adler32 (adler32-zlib) [11828.772535] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11829.094022] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [11830.743406] Key type lgssc registered [11831.871582] Lustre: Echo OBD driver; http://www.lustre.org/ [11840.152411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11840.189877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11841.644768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11847.752795] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11852.261606] LustreError: 358985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11852.280830] LustreError: 358985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11857.377681] LustreError: 358984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.286259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.984511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11868.001797] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11878.734322] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11889.668901] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11898.256456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11898.764152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11906.551392] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11912.156594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11917.637847] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11927.536948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.550659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11927.557967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11927.561340] Lustre: Skipped 2 previous similar messages [11929.588768] Lustre: server umount lustre-MDT0000 complete [11932.641678] LustreError: 358985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.649855] LustreError: 358985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11934.354916] LustreError: 358964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560141 with bad export cookie 11552326970856061368 [11934.386491] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.392970] LustreError: 358964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11935.013243] Lustre: server umount lustre-MDT0001 complete [11951.021941] Lustre: server umount lustre-OST0000 complete [11971.230208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.373456] LustreError: 362700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.426852] LustreError: 362700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11972.680358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11977.692895] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11977.697531] LustreError: 362701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.815456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.946422] LustreError: 362701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.975240] LustreError: 362701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11988.191456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11993.190163] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12002.231218] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12008.217642] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12016.160717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12016.704372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12017.738499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12023.750320] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12032.170434] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12048.895120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.903859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12048.938820] Lustre: Skipped 1 previous similar message [12048.956832] Lustre: Skipped 4 previous similar messages [12051.617349] Lustre: server umount lustre-MDT0000 complete [12053.992927] LustreError: 362700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.017859] LustreError: 362700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12056.419096] LustreError: 362680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560263 with bad export cookie 11552326970856062481 [12056.432191] LustreError: 362680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12056.439249] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12056.809293] Lustre: server umount lustre-MDT0001 complete [12074.499676] Lustre: server umount lustre-OST0000 complete [12093.900178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.421401] LustreError: 366614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.486588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12098.736810] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12107.255903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.472478] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12119.984576] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12126.321894] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12133.464184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12133.738442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12133.741342] Lustre: Skipped 1 previous similar message [12138.874160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12142.234654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12151.637464] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12163.560356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12163.561603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12163.564628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12163.567929] Lustre: Skipped 2 previous similar messages [12168.688455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12168.698844] Lustre: Skipped 1 previous similar message [12173.805656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.823497] Lustre: Skipped 2 previous similar messages [12174.306112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12174.792427] Lustre: server umount lustre-OST0000 complete [12184.034914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12184.051986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12184.059300] Lustre: Skipped 1 previous similar message [12186.716600] Lustre: server umount lustre-MDT0000 complete [12189.153982] LustreError: 366615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.179402] LustreError: 366615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12192.322554] LustreError: 366596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560399 with bad export cookie 11552326970856063503 [12192.342013] LustreError: 366596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12192.346976] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12193.106882] Lustre: server umount lustre-MDT0001 complete [12212.918110] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12215.629503] Key type lgssc unregistered [12215.855750] LNet: 370172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12215.869807] LNetError: 370172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12215.895108] LNet: Removed LNI 192.168.201.127@tcp [12216.969133] Key type .llcrypt unregistered [12216.971053] Key type ._llcrypt unregistered [12244.246371] Key type ._llcrypt registered [12244.248798] Key type .llcrypt registered [12244.344243] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12245.242574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12245.259734] alg: No test for adler32 (adler32-zlib) [12246.461068] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [12246.703208] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12248.351215] Key type lgssc registered [12249.339792] Lustre: Echo OBD driver; http://www.lustre.org/ [12257.803658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12257.834128] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12257.859901] LustreError: 371613:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12257.902210] LustreError: 371613:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12257.918409] LustreError: 371613:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12257.936633] LustreError: 371613:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12257.958305] LustreError: 371613:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12267.466726] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12267.474194] LustreError: 372011:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12267.494526] LustreError: 372011:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12267.500229] LustreError: 372011:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12267.512580] LustreError: 372011:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12267.518046] LustreError: 372011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12290.050576] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12292.750867] Key type lgssc unregistered [12293.061996] LNet: 373016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12293.065601] LNetError: 373016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12293.074927] LNet: Removed LNI 192.168.201.127@tcp [12293.848550] Key type .llcrypt unregistered [12293.851640] Key type ._llcrypt unregistered [12311.916782] Key type ._llcrypt registered [12311.924493] Key type .llcrypt registered [12312.091201] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [12327.004157] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12328.354751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12328.382089] alg: No test for adler32 (adler32-zlib) [12329.534486] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [12329.858239] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12331.504678] Key type lgssc registered [12333.116203] Lustre: Echo OBD driver; http://www.lustre.org/ [12371.177992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12371.219042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12372.504798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12372.542249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12372.632915] Lustre: lustre-MDT0000: new disk, initializing [12372.762774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12372.798085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12377.531823] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12389.755694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12389.931169] Lustre: 376181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12390.039752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12390.045749] Lustre: Skipped 1 previous similar message [12390.133816] Lustre: lustre-MDT0001: new disk, initializing [12390.261446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12390.315140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12390.337990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12396.089811] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12406.873681] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12413.544995] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12420.857936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12421.338267] Lustre: lustre-OST0000: new disk, initializing [12421.342207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12421.351198] Lustre: 378148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12421.514100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12429.027216] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12430.451329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12430.460574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12430.662728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12440.172957] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12445.557656] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12445.805950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12450.404186] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12450.689716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12456.418742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12456.420210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12456.433084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12456.467201] Lustre: Skipped 1 previous similar message [12461.545231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12461.548029] Lustre: Skipped 1 previous similar message [12466.659509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12466.667519] Lustre: Skipped 1 previous similar message [12467.167198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12467.387258] Lustre: server umount lustre-OST0000 complete [12471.777683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.787442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12471.787997] Lustre: Skipped 1 previous similar message [12471.802752] Lustre: Skipped 1 previous similar message [12476.906267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12476.914845] Lustre: Skipped 1 previous similar message [12477.515684] Lustre: server umount lustre-MDT0000 complete [12481.156514] LustreError: 376985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560688 with bad export cookie 11066271212395745219 [12481.157650] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12481.164750] LustreError: 376985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12481.555879] Lustre: server umount lustre-MDT0001 complete [12490.212781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12491.915787] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:38:19 (1787560699) [12498.854729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12499.318520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12503.987841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12504.555944] LustreError: 380609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12504.576295] LustreError: 380609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12508.642342] LustreError: 380608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12513.503287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12513.762024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12514.072660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12519.476700] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12527.422508] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12533.561730] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12540.365073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12540.722715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12546.989520] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12554.934729] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12563.024227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12563.169895] Lustre: lustre-OST0001: new disk, initializing [12563.173525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12563.179579] Lustre: 383444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12563.272167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12570.186461] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12570.204720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12570.212421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12570.336438] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12579.924440] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12585.952295] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12585.959643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12585.967378] LustreError: Skipped 1 previous similar message [12585.970783] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12585.982452] Lustre: Skipped 1 previous similar message [12590.656560] Lustre: server umount lustre-OST0001 complete [12591.079909] LustreError: 382429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12591.116912] LustreError: 382429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12596.198968] LustreError: 382431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12596.237930] LustreError: 382431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12601.314570] LustreError: 382432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.344240] LustreError: 382432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12610.530203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12610.540694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.557899] LustreError: 382431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.584525] LustreError: 382431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12615.267428] Lustre: server umount lustre-OST0000 complete [12621.792289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12621.795222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12621.802743] LustreError: Skipped 1 previous similar message [12621.807674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12621.827653] Lustre: Skipped 2 previous similar messages [12621.841940] Lustre: Skipped 7 previous similar messages [12625.296251] Lustre: server umount lustre-MDT0000 complete [12629.370151] LustreError: 380589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560836 with bad export cookie 11066271212395746101 [12629.377309] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12629.379845] LustreError: 380589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12635.994852] Lustre: server umount lustre-MDT0001 complete [12649.058780] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12651.730753] Key type lgssc unregistered [12652.098312] LNet: 385435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12652.111988] LNetError: 385435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12652.140342] LNet: Removed LNI 192.168.201.127@tcp [12653.182977] Key type .llcrypt unregistered [12653.184401] Key type ._llcrypt unregistered [12706.345349] Key type ._llcrypt registered [12706.347384] Key type .llcrypt registered [12706.442140] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12707.956901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12708.007696] alg: No test for adler32 (adler32-zlib) [12709.057206] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [12709.258405] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12710.919193] Key type lgssc registered [12712.108954] Lustre: Echo OBD driver; http://www.lustre.org/ [12719.809253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12719.837824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12721.135784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12721.170039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12721.412895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12727.434823] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12738.011531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12738.070554] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12738.097987] Lustre: 388238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12738.375941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12743.516146] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12752.130298] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12758.871552] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12764.133383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12764.140509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.159857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.664890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.666138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.448860] Lustre: server umount lustre-MDT0000 complete [12771.078944] LustreError: 388230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560978 with bad export cookie 3919704409291232079 [12771.082848] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12771.089883] LustreError: 388230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12771.370196] Lustre: server umount lustre-MDT0001 complete [12781.101576] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:43:08 (1787560988) [12787.419737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12787.780320] LustreError: 390748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12787.793733] LustreError: 390748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12787.846604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12792.840756] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12793.316435] LustreError: 390749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12798.442772] LustreError: 390748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12801.978309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12802.521210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12807.925533] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12816.374456] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12824.409343] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12832.969623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12833.218939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12833.474350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12840.260570] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12848.833255] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12857.463736] Lustre: Permanently deactivating lustre-OST0000 [12857.476820] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12864.552572] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12867.355553] LustreError: 393737:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12867.368240] LustreError: 393737:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12867.373846] LustreError: 393737:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12870.961290] LustreError: 393786:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12870.965200] LustreError: 393786:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12879.093990] Lustre: server umount lustre-OST0000 complete [12882.103408] LustreError: 394035:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12882.109567] LustreError: 394035:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12888.544730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12888.547539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12888.557121] Lustre: Skipped 1 previous similar message [12888.560342] Lustre: Skipped 1 previous similar message [12893.666248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12893.676055] Lustre: Skipped 1 previous similar message [12896.770719] Lustre: server umount lustre-MDT0000 complete [12898.792530] LustreError: 390749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.818182] LustreError: 390749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12901.003798] LustreError: 390729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561108 with bad export cookie 3919704409291232653 [12901.011573] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12901.021581] LustreError: 390729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12910.755684] Lustre: server umount lustre-MDT0001 complete [12919.969437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12925.388400] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12928.599756] LustreError: 395194:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12935.867884] Lustre: 395533:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.127@tcp [12938.333379] Lustre: server umount MGS complete [12951.125800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12951.731683] LustreError: 396259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12951.762301] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12951.773586] Lustre: Skipped 1 previous similar message [12951.877901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12956.466929] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12962.283362] LustreError: 396260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12962.328874] LustreError: 396260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12967.218444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12967.635249] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12972.690460] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12981.080324] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12987.374757] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12989.482241] Lustre: Permanently reactivating lustre-OST0000 [12989.497876] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12989.504885] Lustre: Skipped 2 previous similar messages [13020.138383] LustreError: 398087:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13028.460937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13028.807286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13028.811213] Lustre: Skipped 1 previous similar message [13035.156181] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13043.650851] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13049.965517] Lustre: server umount lustre-OST0000 complete [13054.947236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13054.955233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.975387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13065.503994] Lustre: 387559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787561257/real 1787561257] req@00000000a1618b3a x1874393475155200/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787561273 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13065.524501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13065.532919] Lustre: Skipped 1 previous similar message [13065.540744] LustreError: 396255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13065.552511] LustreError: 396255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13065.723832] Lustre: server umount lustre-MDT0000 complete [13069.207433] LustreError: 396239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561276 with bad export cookie 3919704409291233710 [13069.209230] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13069.225893] LustreError: 396239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13069.619405] Lustre: server umount lustre-MDT0001 complete [13081.599363] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13083.923321] Key type lgssc unregistered [13084.229403] LNet: 400725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13084.237220] LNetError: 400725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13084.259659] LNet: Removed LNI 192.168.201.127@tcp [13085.107451] Key type .llcrypt unregistered [13085.108781] Key type ._llcrypt unregistered [13105.728965] Key type ._llcrypt registered [13105.731924] Key type .llcrypt registered [13105.828791] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [13127.100846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13128.220156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13128.232964] alg: No test for adler32 (adler32-zlib) [13129.243915] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13129.462753] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13131.143634] Key type lgssc registered [13132.153265] Lustre: Echo OBD driver; http://www.lustre.org/ [13168.260946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13168.298108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.730857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13169.813471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13169.969768] Lustre: lustre-MDT0000: new disk, initializing [13170.213266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13170.250398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13177.000407] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13190.112685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.244736] Lustre: 403887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13190.285545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13190.288880] Lustre: Skipped 1 previous similar message [13190.366464] Lustre: lustre-MDT0001: new disk, initializing [13190.508899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13190.573402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13190.581743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13195.008880] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13205.319773] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13212.628662] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13220.121244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13220.494794] Lustre: lustre-OST0000: new disk, initializing [13220.498864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13220.504588] Lustre: 405855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13220.636591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13227.351813] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13230.705572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13230.719913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13230.841208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13238.996628] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13244.674722] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13244.838686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13249.498667] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13249.699940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13256.672295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13256.685047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.688670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13256.727681] Lustre: Skipped 1 previous similar message [13257.868548] Lustre: server umount lustre-OST0000 complete [13262.304947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13262.309975] LustreError: Skipped 1 previous similar message [13262.311793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.322228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13262.329292] Lustre: Skipped 1 previous similar message [13266.912471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13266.914464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13266.934362] Lustre: Skipped 1 previous similar message [13268.515896] Lustre: server umount lustre-MDT0000 complete [13272.966065] LustreError: 403878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561480 with bad export cookie 5028654657649682376 [13272.979430] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13272.984578] LustreError: 403878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13273.366526] Lustre: server umount lustre-MDT0001 complete [13284.750427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13286.721855] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:51:33 (1787561493) [13296.317116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13296.957736] LustreError: 408319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13296.982638] LustreError: 408319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13297.087530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13302.246120] LustreError: 408320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.259530] LustreError: 408320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13303.032997] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13307.364460] LustreError: 408319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13312.483438] LustreError: 408320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13312.619966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13313.134814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13317.614862] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13325.975575] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13332.128707] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13339.720722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13340.327797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13347.128085] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13355.554202] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13357.996749] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13360.511828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13368.298883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13368.300386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13368.326371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13372.148552] Lustre: server umount lustre-OST0000 complete [13378.529166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13378.541152] Lustre: Skipped 1 previous similar message [13378.546958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13378.554610] Lustre: Skipped 2 previous similar messages [13387.746430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13387.752420] Lustre: Skipped 3 previous similar messages [13390.303176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13390.506939] Lustre: server umount lustre-MDT0000 complete [13392.864377] LustreError: 408316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13392.886254] LustreError: 408316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13394.586880] LustreError: 408300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561602 with bad export cookie 5028654657649683265 [13394.590254] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13394.597818] LustreError: 408300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13394.859477] Lustre: server umount lustre-MDT0001 complete [13405.333716] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13408.094745] Key type lgssc unregistered [13408.361326] LNet: 412051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13408.368901] LNetError: 412051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13408.387142] LNet: Removed LNI 192.168.201.127@tcp [13409.245689] Key type .llcrypt unregistered [13409.247590] Key type ._llcrypt unregistered [13435.960559] Key type ._llcrypt registered [13435.963218] Key type .llcrypt registered [13436.075503] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13436.346548] Key type .llcrypt unregistered [13436.349541] Key type ._llcrypt unregistered [13455.830391] Key type ._llcrypt registered [13455.832063] Key type .llcrypt registered [13455.993054] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13457.653294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13457.686706] alg: No test for adler32 (adler32-zlib) [13458.796912] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13459.105542] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13460.919139] Key type lgssc registered [13462.246449] Lustre: Echo OBD driver; http://www.lustre.org/ [13470.094765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13470.110961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13471.677509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13475.898682] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13481.957952] LustreError: 414268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13481.974254] LustreError: 414268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13485.154573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13485.496660] LustreError: 414269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13485.534667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13490.690793] LustreError: 414263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13490.704717] LustreError: 414263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13493.734589] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13495.783158] LustreError: 414265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13495.825091] LustreError: 414265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13496.847040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13501.932526] LustreError: 414269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13501.967477] LustreError: 414269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13502.581605] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13510.118456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13510.492889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13515.072522] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13525.989155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13525.989302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13525.999991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13526.013831] Lustre: Skipped 1 previous similar message [13529.693617] Lustre: server umount lustre-OST0000 complete [13536.225419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13536.226188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13536.255490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13536.270233] Lustre: Skipped 2 previous similar messages [13540.530700] Lustre: server umount lustre-MDT0000 complete [13541.347294] LustreError: 414269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.364832] LustreError: 414269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13544.642762] LustreError: 414248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561752 with bad export cookie 15857128478543766957 [13544.653439] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13544.656480] LustreError: 414248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13545.212943] Lustre: server umount lustre-MDT0001 complete [13557.161499] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13560.038182] Key type lgssc unregistered [13560.286791] LNet: 417422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13560.295877] LNetError: 417422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13561.325482] LNet: Removed LNI 192.168.201.127@tcp [13562.217161] Key type .llcrypt unregistered [13562.219474] Key type ._llcrypt unregistered [13582.152261] Key type ._llcrypt registered [13582.153895] Key type .llcrypt registered [13582.229443] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13583.237579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13583.284280] alg: No test for adler32 (adler32-zlib) [13584.323749] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13584.527957] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13586.167248] Key type lgssc registered [13587.098834] Lustre: Echo OBD driver; http://www.lustre.org/ [13593.251908] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13593.275198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13608.735381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13613.855371] LustreError: 418792:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [13640.549719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13648.789132] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13658.009600] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13666.309424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13667.139351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13667.209054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13671.549213] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13672.416205] LustreError: 418818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13672.436169] LustreError: 418818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13674.471174] LustreError: 419830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13674.490064] LustreError: 419830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13679.094853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13679.432725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13683.108734] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13689.375967] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13694.035479] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13702.479514] Lustre: server umount lustre-OST0000 complete [13708.800595] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13708.823399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13708.852565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.304785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.307637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13715.430730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13715.441545] Lustre: Skipped 2 previous similar messages [13719.553305] Lustre: server umount lustre-MDT0000 complete [13720.543138] Lustre: 418128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787561912/real 1787561912] req@00000000e982997f x1874394393637504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787561928 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13720.554486] LustreError: 419835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13720.624616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.686216] LustreError: 419835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13724.560803] LustreError: 418799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561932 with bad export cookie 17942806774893130947 [13724.565722] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13724.580496] LustreError: 418799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13724.976134] Lustre: server umount lustre-MDT0001 complete [13739.541307] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13743.044831] Key type lgssc unregistered [13743.461276] LNet: 422367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13743.476994] LNetError: 422367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13744.487817] LNet: Removed LNI 192.168.201.127@tcp [13745.324524] Key type .llcrypt unregistered [13745.325883] Key type ._llcrypt unregistered [13775.314369] Key type ._llcrypt registered [13775.319694] Key type .llcrypt registered [13775.461623] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13776.575922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13776.589836] alg: No test for adler32 (adler32-zlib) [13777.798878] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13778.071141] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13779.807782] Key type lgssc registered [13780.907963] Lustre: Echo OBD driver; http://www.lustre.org/ [13789.434799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13789.451145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13790.996653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13795.315802] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13801.442957] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13801.462290] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13803.321949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13803.568726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13807.236411] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13813.586233] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13818.203608] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13824.695486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13825.050345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13830.468455] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13836.335188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13837.425278] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13844.448625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13844.457969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13844.478978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13845.490182] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [13846.499863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13846.500243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13846.517763] Lustre: Skipped 1 previous similar message [13846.534359] Lustre: Skipped 2 previous similar messages [13848.455175] Lustre: server umount lustre-MDT0000 complete [13850.684391] LustreError: 423773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.27@tcp arrived at 1787562058 with bad export cookie 16724114806382903686 [13851.619950] LustreError: 424855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13851.639110] LustreError: 424855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13852.264550] LustreError: 423774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562059 with bad export cookie 16724114806382902790 [13852.276734] LustreError: 423774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13852.288635] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13854.688575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13854.697419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13854.711871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13856.736524] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.764986] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13858.506912] Lustre: server umount lustre-OST0000 complete [13861.861221] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.869845] LustreError: 423793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13864.463972] Lustre: server umount lustre-MDT0001 complete [13877.275408] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13879.963674] Key type lgssc unregistered [13880.192934] LNet: 427488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13880.199237] LNetError: 427488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13880.208961] LNet: Removed LNI 192.168.201.127@tcp [13880.901606] Key type .llcrypt unregistered [13880.904806] Key type ._llcrypt unregistered [13904.346727] Key type ._llcrypt registered [13904.348564] Key type .llcrypt registered [13904.458174] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13905.381424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13905.490511] alg: No test for adler32 (adler32-zlib) [13906.584138] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13906.778437] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13908.399867] Key type lgssc registered [13909.318219] Lustre: Echo OBD driver; http://www.lustre.org/ [13916.543735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13916.579773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13918.004784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13921.887162] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13928.429229] LustreError: 428903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13928.451880] LustreError: 428903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13929.965243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13930.218235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13934.337326] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13941.993486] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13947.669752] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13954.229896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13954.596903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13961.034437] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13962.831392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13968.332930] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13976.547487] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13976.552113] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13976.574803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13976.587208] Lustre: Skipped 1 previous similar message [13978.083578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13978.085968] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13979.243597] Lustre: server umount lustre-MDT0001 complete [13983.202465] LustreError: 429631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13983.224055] LustreError: 429631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13988.322144] LustreError: 428904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13988.332992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13988.334693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13988.347092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13988.349725] Lustre: Skipped 1 previous similar message [13991.894350] Lustre: server umount lustre-OST0000 complete [13993.449361] LustreError: 429631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13995.535264] Lustre: server umount lustre-MDT0000 complete [14010.205290] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14012.532586] Key type lgssc unregistered [14012.748356] LNet: 432588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14012.757158] LNetError: 432588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14012.774861] LNet: Removed LNI 192.168.201.127@tcp [14013.634142] Key type .llcrypt unregistered [14013.637040] Key type ._llcrypt unregistered [14040.434683] Key type ._llcrypt registered [14040.442717] Key type .llcrypt registered [14040.551405] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14056.276482] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14057.668782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14057.698680] alg: No test for adler32 (adler32-zlib) [14058.710443] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [14058.936466] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [14060.615322] Key type lgssc registered [14062.130684] Lustre: Echo OBD driver; http://www.lustre.org/ [14106.278673] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14126.390483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14126.455293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14127.900331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14127.991132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14128.185307] Lustre: lustre-MDT0000: new disk, initializing [14128.362255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14128.465880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14135.705869] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14153.997501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14154.410978] Lustre: lustre-OST0000: new disk, initializing [14154.420936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14154.423284] Lustre: Skipped 1 previous similar message [14154.433797] Lustre: 437630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14154.537533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14161.336746] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14161.512776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14161.527957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14161.543682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14171.194805] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14187.714900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14187.863117] Lustre: 436609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14187.934086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14188.002608] Lustre: lustre-MDT0001: new disk, initializing [14188.110222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14188.143498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14188.154947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14192.785598] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14202.704877] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14202.918564] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14208.451309] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14208.821496] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14214.124687] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [14218.212490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14218.212918] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14218.216883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14218.235170] Lustre: Skipped 1 previous similar message [14218.262385] Lustre: Skipped 1 previous similar message [14220.419923] Lustre: server umount lustre-MDT0001 complete [14228.450502] LustreError: 436621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14228.470584] LustreError: 436621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14233.597052] LustreError: 439329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14233.624715] LustreError: 439329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14238.695763] LustreError: 436621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14243.301637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14243.308282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14243.323139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14243.329744] Lustre: Skipped 1 previous similar message [14243.810387] LustreError: 436622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14244.506885] Lustre: server umount lustre-OST0000 complete [14254.815287] Lustre: server umount lustre-MDT0000 complete [14269.760857] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14272.434448] Key type lgssc unregistered [14272.853814] LNet: 441379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14272.861816] LNetError: 441379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14272.879095] LNet: Removed LNI 192.168.201.127@tcp [14273.755397] Key type .llcrypt unregistered [14273.756537] Key type ._llcrypt unregistered [14316.546602] Key type ._llcrypt registered [14316.548519] Key type .llcrypt registered [14316.670182] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14316.977567] Key type .llcrypt unregistered [14316.979215] Key type ._llcrypt unregistered [14333.729248] Key type ._llcrypt registered [14333.734029] Key type .llcrypt registered [14333.902868] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14335.755330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14335.797146] alg: No test for adler32 (adler32-zlib) [14336.806379] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [14337.031400] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [14338.727191] Key type lgssc registered [14339.801604] Lustre: Echo OBD driver; http://www.lustre.org/ [14370.034145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14370.065799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.510991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14371.573489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14371.766252] Lustre: lustre-MDT0000: new disk, initializing [14371.867300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14371.894766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14377.711665] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14390.389487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14390.631870] Lustre: 444760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14390.734494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14390.738929] Lustre: Skipped 1 previous similar message [14390.906816] Lustre: lustre-MDT0001: new disk, initializing [14391.073935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14391.162078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14391.184895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14395.537503] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14405.880167] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14411.999744] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14419.414791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14419.638667] Lustre: lustre-OST0000: new disk, initializing [14419.647744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14419.659624] Lustre: 446731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14419.777718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14427.029652] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14427.769908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14427.782341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14427.938414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14437.658607] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14447.585444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14447.589096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14447.601706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14448.610461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14448.613606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14448.639844] Lustre: Skipped 1 previous similar message [14448.666762] Lustre: Skipped 2 previous similar messages [14450.644951] Lustre: server umount lustre-MDT0000 complete [14454.521918] LustreError: 446730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562662 with bad export cookie 4595269012048413719 [14454.526657] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14454.540159] LustreError: 446730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14455.060856] Lustre: server umount lustre-MDT0001 complete [14469.591542] Lustre: server umount lustre-OST0000 complete [14481.585548] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:11:28 (1787562688) [14489.647510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14490.306840] LustreError: 448760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14490.326979] LustreError: 448760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14490.416608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14495.713125] LustreError: 448761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14495.959977] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14499.820283] LustreError: 448760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14504.929267] LustreError: 448761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14505.492051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14505.909521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14511.082252] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14519.691616] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14526.428596] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14546.544918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14546.783560] Lustre: Found index 0 for lustre-OST0000, updating log [14546.983910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14552.827863] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14561.020722] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14561.354789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14566.373238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14566.386499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14566.397896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14566.408530] LustreError: Skipped 1 previous similar message [14567.741528] Lustre: server umount lustre-OST0000 complete [14587.872245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14587.887581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14587.902861] Lustre: Skipped 1 previous similar message [14587.918129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14587.922482] Lustre: Skipped 3 previous similar messages [14590.366369] Lustre: server umount lustre-MDT0000 complete [14594.017173] LustreError: 449488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14594.038531] LustreError: 449488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14595.435367] LustreError: 448742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562802 with bad export cookie 4595269012048415112 [14595.442343] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14595.445709] LustreError: 448742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14595.826264] Lustre: server umount lustre-MDT0001 complete [14607.681339] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:13:34 (1787562814) [14613.599571] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [15236.533986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15237.384036] LustreError: 455171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15237.408254] LustreError: 455171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15237.509471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15242.720076] LustreError: 455172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15242.898403] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15246.821487] LustreError: 455171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15252.966976] LustreError: 455172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15252.998889] LustreError: 455172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15253.648838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15254.056004] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15259.576702] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15269.158750] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15275.851528] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15291.487808] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [15297.847517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15298.335464] Lustre: Found index 0 for lustre-OST0000, updating log [15298.768502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15305.381830] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15313.999530] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15317.490567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15317.504522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15317.524460] Lustre: Skipped 1 previous similar message [15317.533856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15317.544640] Lustre: Skipped 2 previous similar messages [15323.000024] Lustre: server umount lustre-OST0000 complete [15328.481315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15328.508500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15328.523349] Lustre: Skipped 1 previous similar message [15329.571033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15329.573547] Lustre: Skipped 1 previous similar message [15331.683822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15331.688455] Lustre: Skipped 3 previous similar messages [15333.738397] LustreError: 455171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15333.917889] Lustre: server umount lustre-MDT0000 complete [15338.188799] LustreError: 455151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563545 with bad export cookie 4595269012048416197 [15338.201487] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15338.204113] LustreError: 455151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15338.659718] Lustre: server umount lustre-MDT0001 complete [15351.903299] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [15371.627873] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [15383.961211] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15431.045916] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:27:17 (1787563637) [15453.209972] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15456.119643] Key type lgssc unregistered [15456.421796] LNet: 461790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15456.434264] LNetError: 461790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15557.865396] LNet: Removed LNI 192.168.201.127@tcp [15558.917455] Key type .llcrypt unregistered [15558.921507] Key type ._llcrypt unregistered [15580.968160] Key type ._llcrypt registered [15580.970879] Key type .llcrypt registered [15581.066505] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15582.446384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15582.771802] alg: No test for adler32 (adler32-zlib) [15584.269390] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [15584.726083] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15586.487239] Key type lgssc registered [15587.839919] Lustre: Echo OBD driver; http://www.lustre.org/ [15594.220789] LNet: Added LNI 192.168.201.127@tcp100 [8/256/0/180] [15595.511597] LNet: Added LNI 192.168.201.127@tcp101 [8/256/0/180] [15596.656439] LNet: Added LNI 192.168.201.127@tcp102 [8/256/0/180] [15597.740775] LNet: Added LNI 192.168.201.127@tcp103 [8/256/0/180] [15598.693042] LNet: Added LNI 192.168.201.127@tcp104 [8/256/0/180] [15599.851957] LNet: Added LNI 192.168.201.127@tcp105 [8/256/0/180] [15600.990279] LNet: Added LNI 192.168.201.127@tcp106 [8/256/0/180] [15602.164617] LNet: Added LNI 192.168.201.127@tcp107 [8/256/0/180] [15603.226892] LNet: Added LNI 192.168.201.127@tcp108 [8/256/0/180] [15604.559728] LNet: Added LNI 192.168.201.127@tcp109 [8/256/0/180] [15605.595910] LNet: Added LNI 192.168.201.127@tcp110 [8/256/0/180] [15606.679180] LNet: Added LNI 192.168.201.127@tcp111 [8/256/0/180] [15607.771862] LNet: Added LNI 192.168.201.127@tcp112 [8/256/0/180] [15608.830544] LNet: Added LNI 192.168.201.127@tcp113 [8/256/0/180] [15609.901505] LNet: Added LNI 192.168.201.127@tcp114 [8/256/0/180] [15610.760647] LNet: Added LNI 192.168.201.127@tcp115 [8/256/0/180] [15612.034282] LNet: Added LNI 192.168.201.127@tcp116 [8/256/0/180] [15613.066763] LNet: Added LNI 192.168.201.127@tcp117 [8/256/0/180] [15614.473144] LNet: Added LNI 192.168.201.127@tcp118 [8/256/0/180] [15615.488660] LNet: Added LNI 192.168.201.127@tcp119 [8/256/0/180] [15616.811218] LNet: Added LNI 192.168.201.127@tcp120 [8/256/0/180] [15618.143507] LNet: Added LNI 192.168.201.127@tcp121 [8/256/0/180] [15619.150769] LNet: Added LNI 192.168.201.127@tcp122 [8/256/0/180] [15620.332296] LNet: Added LNI 192.168.201.127@tcp123 [8/256/0/180] [15621.503486] LNet: Added LNI 192.168.201.127@tcp124 [8/256/0/180] [15622.771993] LNet: Added LNI 192.168.201.127@tcp125 [8/256/0/180] [15624.084252] LNet: Added LNI 192.168.201.127@tcp126 [8/256/0/180] [15625.189746] LNet: Added LNI 192.168.201.127@tcp127 [8/256/0/180] [15626.224577] LNet: Added LNI 192.168.201.127@tcp128 [8/256/0/180] [15627.221448] LNet: Added LNI 192.168.201.127@tcp129 [8/256/0/180] [15628.269300] LNet: Added LNI 192.168.201.127@tcp130 [8/256/0/180] [15628.993238] LNet: Added LNI 192.168.201.127@tcp131 [8/256/0/180] [15629.678979] LNet: Added LNI 192.168.201.127@tcp132 [8/256/0/180] [15630.630862] LNet: Added LNI 192.168.201.127@tcp133 [8/256/0/180] [15631.636866] LNet: Added LNI 192.168.201.127@tcp134 [8/256/0/180] [15632.709124] LNet: Added LNI 192.168.201.127@tcp135 [8/256/0/180] [15640.684949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15640.723082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15641.834744] Lustre: 464823:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [15641.943871] LNetError: 464847:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15641.953306] LNetError: Protocol error connecting to 192.168.201.127@tcp122 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15641.967800] LNetError: Skipped 6 previous similar messages [15642.083154] LNet: 5 local NIs in recovery (showing 5): 192.168.201.127@tcp125, 192.168.201.127@tcp126, 192.168.201.127@tcp127, 192.168.201.127@tcp128, 192.168.201.127@tcp122 [15642.113710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15642.144302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15642.285144] Lustre: lustre-MDT0000: new disk, initializing [15642.352957] Lustre: 464881:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [15642.372059] Lustre: 464881:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15642.401448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15642.438823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15644.129091] LNetError: 462277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15644.142566] LNetError: 462277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15644.151961] LNetError: Protocol error connecting to 192.168.201.127@tcp125 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15644.167830] LNetError: Skipped 7 previous similar messages [15646.945693] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15648.224596] LNetError: 464854:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15648.244173] LNetError: 464854:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15648.249538] LNetError: Protocol error connecting to 192.168.201.127@tcp125 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15656.417115] LNetError: 464847:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15656.438201] LNetError: Protocol error connecting to 192.168.201.127@tcp125 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15658.225732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15658.359740] Lustre: 464841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15658.407355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15658.411120] Lustre: Skipped 1 previous similar message [15658.472346] Lustre: lustre-MDT0001: new disk, initializing [15658.488830] Lustre: 465643:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [15658.491956] Lustre: 465643:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15658.567858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15658.601847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15658.616985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15663.567863] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15665.701616] Lustre: 466007:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [15665.710486] Lustre: 466007:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15672.801115] LNetError: 462278:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15672.825042] LNetError: Protocol error connecting to 192.168.201.127@tcp125 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15673.350465] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15679.700604] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15686.177788] LNetError: 462277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15686.193972] LNetError: 462277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15686.212624] LNetError: Protocol error connecting to 192.168.201.127@tcp127 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15686.223545] LNetError: Skipped 6 previous similar messages [15687.651725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15688.165338] Lustre: lustre-OST0000: new disk, initializing [15688.183583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15688.192095] Lustre: 466814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15688.309488] Lustre: 466819:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [15688.315855] Lustre: 466819:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15688.373829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15693.949096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15693.977422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15694.186877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15697.182729] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15701.473471] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp125 [15703.593934] LNetError: 464851:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15703.616084] LNetError: 464851:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15703.645924] LNetError: Protocol error connecting to 192.168.201.127@tcp120 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15703.666530] LNetError: Skipped 7 previous similar messages [15709.257766] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15719.910924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15719.929643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15719.936514] Lustre: Skipped 1 previous similar message [15719.944612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15719.948107] Lustre: Skipped 1 previous similar message [15725.045080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15725.068942] Lustre: Skipped 1 previous similar message [15730.166947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15730.176338] Lustre: Skipped 1 previous similar message [15731.167187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15731.368831] Lustre: server umount lustre-OST0000 complete [15735.264816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15735.287650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15735.297643] Lustre: Skipped 1 previous similar message [15738.466126] LNetError: 464854:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [15738.477783] LNetError: 464854:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15738.488914] LNetError: Protocol error connecting to 192.168.201.127@tcp122 at host 192.168.201.127:988: is it running a compatible version of Lustre? [15738.501745] LNetError: Skipped 11 previous similar messages [15740.403393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15740.412124] Lustre: Skipped 1 previous similar message [15741.632845] Lustre: server umount lustre-MDT0000 complete [15745.962961] LustreError: 467972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563953 with bad export cookie 12043226406076684781 [15745.969299] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15745.969674] LustreError: 467972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15746.647544] Lustre: server umount lustre-MDT0001 complete [15759.071766] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15762.029256] Key type lgssc unregistered [15762.376941] LNet: 468735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15762.382424] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15763.424298] LNet: Removed LNI 192.168.201.127@tcp [15763.443031] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15763.452549] LNet: Removed LNI 192.168.201.127@tcp100 [15764.511262] LNet: Removed LNI 192.168.201.127@tcp101 [15764.514789] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15764.528459] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15764.537871] LNet: Removed LNI 192.168.201.127@tcp102 [15765.599237] LNet: Removed LNI 192.168.201.127@tcp103 [15766.623213] LNet: Removed LNI 192.168.201.127@tcp104 [15766.629756] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15766.640383] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15766.654451] LNet: Removed LNI 192.168.201.127@tcp105 [15767.715415] LNet: Removed LNI 192.168.201.127@tcp106 [15768.735258] LNet: Removed LNI 192.168.201.127@tcp107 [15769.759224] LNet: Removed LNI 192.168.201.127@tcp108 [15770.783481] LNet: Removed LNI 192.168.201.127@tcp109 [15770.788817] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15770.805432] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15770.820307] LNet: Removed LNI 192.168.201.127@tcp110 [15771.871177] LNet: Removed LNI 192.168.201.127@tcp111 [15772.895303] LNet: Removed LNI 192.168.201.127@tcp112 [15773.919327] LNet: Removed LNI 192.168.201.127@tcp113 [15774.943221] LNet: Removed LNI 192.168.201.127@tcp114 [15775.967213] LNet: Removed LNI 192.168.201.127@tcp115 [15776.991469] LNet: Removed LNI 192.168.201.127@tcp116 [15778.015244] LNet: Removed LNI 192.168.201.127@tcp117 [15779.045450] LNet: Removed LNI 192.168.201.127@tcp118 [15779.053579] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15779.068874] LNetError: 468735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15779.075988] LNet: Removed LNI 192.168.201.127@tcp119 [15780.127523] LNet: Removed LNI 192.168.201.127@tcp120 [15781.151221] LNet: Removed LNI 192.168.201.127@tcp121 [15782.175269] LNet: Removed LNI 192.168.201.127@tcp122 [15783.199154] LNet: Removed LNI 192.168.201.127@tcp123 [15784.223276] LNet: Removed LNI 192.168.201.127@tcp124 [15784.227294] LNet: Removed LNI 192.168.201.127@tcp125 [15785.247334] LNet: Removed LNI 192.168.201.127@tcp126 [15786.271241] LNet: Removed LNI 192.168.201.127@tcp127 [15787.295693] LNet: Removed LNI 192.168.201.127@tcp128 [15788.319685] LNet: Removed LNI 192.168.201.127@tcp129 [15788.325586] LNet: Removed LNI 192.168.201.127@tcp130 [15788.331345] LNet: Removed LNI 192.168.201.127@tcp131 [15788.334584] LNet: Removed LNI 192.168.201.127@tcp132 [15788.338954] LNet: Removed LNI 192.168.201.127@tcp133 [15788.342924] LNet: Removed LNI 192.168.201.127@tcp134 [15788.358701] LNet: Removed LNI 192.168.201.127@tcp135 [15789.134341] Key type .llcrypt unregistered [15789.138067] Key type ._llcrypt unregistered [15811.608560] Key type ._llcrypt registered [15811.610717] Key type .llcrypt registered [15811.691413] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [15827.264979] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15828.695985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15828.762614] alg: No test for adler32 (adler32-zlib) [15829.787362] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [15830.114828] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15831.799224] Key type lgssc registered [15833.086227] Lustre: Echo OBD driver; http://www.lustre.org/ [15870.472402] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:34:37 (1787564077) [15890.446577] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15893.851761] Key type lgssc unregistered [15894.309380] LNet: 472830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15894.331161] LNetError: 472830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15894.360379] LNet: Removed LNI 192.168.201.127@tcp [15895.394952] Key type .llcrypt unregistered [15895.399523] Key type ._llcrypt unregistered [15914.168579] Key type ._llcrypt registered [15914.170616] Key type .llcrypt registered [15914.280763] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15915.265445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15915.429176] alg: No test for adler32 (adler32-zlib) [15916.511402] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [15916.799245] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15918.495391] Key type lgssc registered [15919.603883] Lustre: Echo OBD driver; http://www.lustre.org/ [15929.621769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15929.683403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15931.164499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15931.219490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15931.332634] Lustre: lustre-MDT0000: new disk, initializing [15931.464164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15931.483790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15936.483697] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15948.123073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15948.196566] Lustre: 474042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15948.243093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15948.246837] Lustre: Skipped 1 previous similar message [15948.297192] Lustre: lustre-MDT0001: new disk, initializing [15948.378546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15948.411445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15948.420626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15952.227691] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15961.664256] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15968.080874] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15976.001459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15976.413807] Lustre: lustre-OST0000: new disk, initializing [15976.417152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15976.433978] Lustre: 476011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15976.547328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15982.932687] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15986.257423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15986.280223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15986.478282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15993.460824] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15997.893572] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing lnet_if_list [16006.077371] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16006.166090] LNet: Added LNI 192.168.201.127@tcp10 [8/256/0/180] [16016.544239] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16016.595742] LNet: Added LNI 192.168.201.127@tcp11 [8/256/0/180] [16022.537845] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16022.595785] LNet: Added LNI 192.168.201.127@tcp12 [8/256/0/180] [16027.760522] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16027.813317] LNet: Added LNI 192.168.201.127@tcp13 [8/256/0/180] [16032.471819] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16032.511538] LNet: Added LNI 192.168.201.127@tcp14 [8/256/0/180] [16037.639581] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16037.674237] LNet: Added LNI 192.168.201.127@tcp15 [8/256/0/180] [16042.473560] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16042.521825] LNet: Added LNI 192.168.201.127@tcp16 [8/256/0/180] [16049.022965] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16049.107581] LNet: Added LNI 192.168.201.127@tcp17 [8/256/0/180] [16054.660457] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16054.687676] LNet: Added LNI 192.168.201.127@tcp18 [8/256/0/180] [16061.686138] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16061.737147] LNet: Added LNI 192.168.201.127@tcp19 [8/256/0/180] [16067.043454] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16067.063031] LNet: Added LNI 192.168.201.127@tcp20 [8/256/0/180] [16072.145710] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16072.183269] LNet: Added LNI 192.168.201.127@tcp21 [8/256/0/180] [16077.067887] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16077.105867] LNet: Added LNI 192.168.201.127@tcp22 [8/256/0/180] [16082.392383] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16082.417894] LNet: Added LNI 192.168.201.127@tcp23 [8/256/0/180] [16089.104488] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16089.157751] LNet: Added LNI 192.168.201.127@tcp24 [8/256/0/180] [16096.419638] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16096.444842] LNet: Added LNI 192.168.201.127@tcp25 [8/256/0/180] [16101.818283] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16101.842990] LNet: Added LNI 192.168.201.127@tcp26 [8/256/0/180] [16108.277501] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16108.302181] LNet: Added LNI 192.168.201.127@tcp27 [8/256/0/180] [16114.063819] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16114.088782] LNet: Added LNI 192.168.201.127@tcp28 [8/256/0/180] [16120.029873] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16120.102080] LNet: Added LNI 192.168.201.127@tcp29 [8/256/0/180] [16126.676693] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16126.733733] LNet: Added LNI 192.168.201.127@tcp30 [8/256/0/180] [16132.896964] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16132.925936] LNet: Added LNI 192.168.201.127@tcp31 [8/256/0/180] [16138.873464] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16138.899791] LNet: Added LNI 192.168.201.127@tcp32 [8/256/0/180] [16144.985528] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16145.009668] LNet: Added LNI 192.168.201.127@tcp33 [8/256/0/180] [16150.809605] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16150.829552] LNet: Added LNI 192.168.201.127@tcp34 [8/256/0/180] [16156.368400] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16156.389287] LNet: Added LNI 192.168.201.127@tcp35 [8/256/0/180] [16161.997793] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16162.025322] LNet: Added LNI 192.168.201.127@tcp36 [8/256/0/180] [16168.281575] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16168.310754] LNet: Added LNI 192.168.201.127@tcp37 [8/256/0/180] [16175.525430] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16175.549896] LNet: Added LNI 192.168.201.127@tcp38 [8/256/0/180] [16182.540820] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16182.607958] LNet: Added LNI 192.168.201.127@tcp39 [8/256/0/180] [16189.484494] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16189.512928] LNet: Added LNI 192.168.201.127@tcp40 [8/256/0/180] [16196.002414] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16196.027644] LNet: Added LNI 192.168.201.127@tcp41 [8/256/0/180] [16203.307357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16203.355276] LNet: Added LNI 192.168.201.127@tcp42 [8/256/0/180] [16211.085310] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16211.138842] LNet: Added LNI 192.168.201.127@tcp43 [8/256/0/180] [16218.356186] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16218.404027] LNet: Added LNI 192.168.201.127@tcp44 [8/256/0/180] [16226.197582] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16226.218537] LNet: Added LNI 192.168.201.127@tcp45 [8/256/0/180] [16234.661941] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16234.712712] LNet: Added LNI 192.168.201.127@tcp46 [8/256/0/180] [16244.814715] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16244.890223] LNet: Added LNI 192.168.201.127@tcp47 [8/256/0/180] [16256.419135] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16256.509503] LNet: Added LNI 192.168.201.127@tcp48 [8/256/0/180] [16266.930969] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16266.972230] LNet: Added LNI 192.168.201.127@tcp49 [8/256/0/180] [16276.034811] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16276.071446] LNet: Added LNI 192.168.201.127@tcp50 [8/256/0/180] [16296.416891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16296.429864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16296.446941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16298.981478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16298.981598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16298.994097] Lustre: Skipped 1 previous similar message [16299.010746] Lustre: Skipped 2 previous similar messages [16300.984704] Lustre: server umount lustre-MDT0000 complete [16305.282848] LustreError: 474034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564512 with bad export cookie 11960203020514141767 [16305.285309] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16305.308415] LustreError: 474034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16305.734220] Lustre: server umount lustre-MDT0001 complete [16324.577235] Lustre: 473314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787564516/real 1787564516] req@00000000f7c1739d x1874396839060096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787564532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16324.600506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16325.589186] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16325.599347] Lustre: 473315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787564517/real 1787564517] req@00000000e2425ecc x1874396839060224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787564533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16329.696749] Lustre: 473316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787564521/real 1787564521] req@00000000cd21b4b6 x1874396839060352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787564537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16334.927718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16336.865882] Lustre: 473315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787564528/real 1787564528] req@00000000615d8a72 x1874396839060736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787564544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16336.883082] Lustre: 473315:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16350.241548] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa5fb2d477f4cb073 [16350.251727] Lustre: MGC192.168.201.127@tcp: Connection restored to 0@lo (at 0@lo) [16350.570232] LustreError: 476005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16350.593855] LustreError: 476005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16350.652362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16353.698988] LustreError: 476006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16353.710162] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16355.277469] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16355.746832] LustreError: 476005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16360.936787] LustreError: 476006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16364.638736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16365.238374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16370.148962] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16370.378554] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16379.292512] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16386.257803] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16411.111473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16411.134572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16411.176183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16415.759664] Lustre: server umount lustre-OST0000 complete [16421.345252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16421.365366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16421.398940] Lustre: Skipped 1 previous similar message [16421.406531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16421.415730] Lustre: Skipped 1 previous similar message [16425.932392] Lustre: server umount lustre-MDT0000 complete [16426.976387] LustreError: 502117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16427.009440] LustreError: 502117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16430.123500] LustreError: 474035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564637 with bad export cookie 11960203020514144371 [16430.125161] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16430.137815] LustreError: 474035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16430.909410] Lustre: server umount lustre-MDT0001 complete [16444.585171] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [16448.078760] Key type lgssc unregistered [16448.515589] LNet: 503058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16448.528804] LNetError: 503058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16448.534239] LNet: Removed LNI 192.168.201.127@tcp [16448.536610] LNet: Removed LNI 192.168.201.127@tcp10 [16448.540726] LNet: Removed LNI 192.168.201.127@tcp11 [16448.548620] LNet: Removed LNI 192.168.201.127@tcp12 [16448.558414] LNet: Removed LNI 192.168.201.127@tcp13 [16448.566617] LNet: Removed LNI 192.168.201.127@tcp14 [16448.572567] LNet: Removed LNI 192.168.201.127@tcp15 [16448.578303] LNet: Removed LNI 192.168.201.127@tcp16 [16448.581861] LNet: Removed LNI 192.168.201.127@tcp17 [16448.586111] LNet: Removed LNI 192.168.201.127@tcp18 [16448.587903] LNet: Removed LNI 192.168.201.127@tcp19 [16448.595521] LNet: Removed LNI 192.168.201.127@tcp20 [16448.599700] LNet: Removed LNI 192.168.201.127@tcp21 [16448.602878] LNet: Removed LNI 192.168.201.127@tcp22 [16448.611605] LNet: Removed LNI 192.168.201.127@tcp23 [16448.614980] LNet: Removed LNI 192.168.201.127@tcp24 [16448.617939] LNet: Removed LNI 192.168.201.127@tcp25 [16448.620785] LNet: Removed LNI 192.168.201.127@tcp26 [16448.625410] LNet: Removed LNI 192.168.201.127@tcp27 [16448.634593] LNet: Removed LNI 192.168.201.127@tcp28 [16448.638481] LNet: Removed LNI 192.168.201.127@tcp29 [16448.641863] LNet: Removed LNI 192.168.201.127@tcp30 [16448.645398] LNet: Removed LNI 192.168.201.127@tcp31 [16448.648630] LNet: Removed LNI 192.168.201.127@tcp32 [16448.654132] LNet: Removed LNI 192.168.201.127@tcp33 [16448.655630] LNet: Removed LNI 192.168.201.127@tcp34 [16448.663892] LNet: Removed LNI 192.168.201.127@tcp35 [16448.668605] LNet: Removed LNI 192.168.201.127@tcp36 [16448.672572] LNet: Removed LNI 192.168.201.127@tcp37 [16448.676544] LNet: Removed LNI 192.168.201.127@tcp38 [16448.682498] LNet: Removed LNI 192.168.201.127@tcp39 [16448.692321] LNet: Removed LNI 192.168.201.127@tcp40 [16448.694764] LNet: Removed LNI 192.168.201.127@tcp41 [16448.706232] LNet: Removed LNI 192.168.201.127@tcp42 [16448.711099] LNet: Removed LNI 192.168.201.127@tcp43 [16448.713540] LNet: Removed LNI 192.168.201.127@tcp44 [16448.721454] LNet: Removed LNI 192.168.201.127@tcp45 [16448.725321] LNet: Removed LNI 192.168.201.127@tcp46 [16448.727937] LNet: Removed LNI 192.168.201.127@tcp47 [16448.733174] LNet: Removed LNI 192.168.201.127@tcp48 [16448.734923] LNet: Removed LNI 192.168.201.127@tcp49 [16448.743950] LNet: Removed LNI 192.168.201.127@tcp50 [16449.367924] Key type .llcrypt unregistered [16449.372592] Key type ._llcrypt unregistered [16471.637607] Key type ._llcrypt registered [16471.639798] Key type .llcrypt registered [16471.731392] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [16488.302545] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16489.434505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16489.595077] alg: No test for adler32 (adler32-zlib) [16490.678896] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [16490.902887] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [16492.583329] Key type lgssc registered [16494.165652] Lustre: Echo OBD driver; http://www.lustre.org/ [16533.747892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16533.766895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16535.101206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16535.138170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16535.216233] Lustre: lustre-MDT0000: new disk, initializing [16535.294463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16535.309342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16541.128459] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16556.708264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16556.885846] Lustre: 506256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16556.933901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16556.937590] Lustre: Skipped 1 previous similar message [16557.027225] Lustre: lustre-MDT0001: new disk, initializing [16557.142454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16557.176687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16557.200222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16562.517200] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16573.363832] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16579.610271] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16587.719411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16587.931247] Lustre: *** cfs_fail_loc=90f, val=0*** [16588.006908] Lustre: lustre-OST0000: new disk, initializing [16588.017400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16588.024723] Lustre: 508276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16588.099984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16594.991744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16595.004329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16595.143061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16595.698301] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16606.361121] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16615.905375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16615.906744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16615.926675] LustreError: Skipped 1 previous similar message [16615.935450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16615.959188] Lustre: Skipped 1 previous similar message [16615.987675] Lustre: Skipped 1 previous similar message [16621.029722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16621.039081] Lustre: Skipped 1 previous similar message [16626.153324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16626.157953] Lustre: Skipped 1 previous similar message [16629.215335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16629.555853] Lustre: server umount lustre-OST0000 complete [16636.385943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16636.389962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16636.419742] Lustre: Skipped 1 previous similar message [16636.436095] Lustre: Skipped 1 previous similar message [16640.542828] Lustre: server umount lustre-MDT0000 complete [16645.065661] LustreError: 506249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564852 with bad export cookie 3096337458136028886 [16645.066445] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16645.073376] LustreError: 506249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16645.533024] Lustre: server umount lustre-MDT0001 complete [16658.672527] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [16661.957360] Key type lgssc unregistered [16662.365700] LNet: 510189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16662.378661] LNetError: 510189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16662.403500] LNet: Removed LNI 192.168.201.127@tcp [16663.250738] Key type .llcrypt unregistered [16663.253238] Key type ._llcrypt unregistered [16686.399566] Key type ._llcrypt registered [16686.407387] Key type .llcrypt registered [16686.542955] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16688.158531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16688.214793] alg: No test for adler32 (adler32-zlib) [16689.303977] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [16689.628151] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [16691.319713] Key type lgssc registered [16692.588728] Lustre: Echo OBD driver; http://www.lustre.org/ [16700.707550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16700.732341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16702.232207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16707.356616] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16711.660271] LustreError: 511467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16711.701290] LustreError: 511467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16716.785996] LustreError: 511466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16717.133926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16717.575943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16722.956557] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16732.468554] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16738.585347] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16745.725410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16746.250300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16754.352708] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16761.256507] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16772.829513] loop0: detected capacity change from 0 to 204800000 [16773.060693] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16773.072395] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16773.109524] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16781.456320] loop0: detected capacity change from 0 to 204800000 [16781.716028] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16781.723817] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16781.741362] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16789.073417] loop0: detected capacity change from 0 to 204800000 [16794.980107] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16795.149357] Lustre: fs15246-OST0000: new disk, initializing [16795.153158] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16795.162175] Lustre: 515061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16795.235817] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16800.758467] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16810.706845] loop1: detected capacity change from 0 to 204800000 [16817.098669] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16817.182556] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16817.232957] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16817.292256] Lustre: fs15246-MDT0000: new disk, initializing [16817.474934] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16817.502882] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16817.652298] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16818.724220] Lustre: 516184:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000020200671 x1874397649485952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787565042 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16822.296714] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16823.862411] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16823.872329] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16823.903416] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16849.381259] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16849.393236] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16851.256753] Lustre: server umount fs15246-MDT0000 complete [16854.510261] LustreError: 511461:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16859.636240] LustreError: 511462:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16862.100891] Lustre: server umount fs15246-OST0000 complete [16875.490777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16875.494106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16875.502836] LustreError: Skipped 1 previous similar message [16875.504756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16875.527478] Lustre: Skipped 1 previous similar message [16875.551289] Lustre: Skipped 1 previous similar message [16880.097713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16880.115563] Lustre: Skipped 1 previous similar message [16881.735577] Lustre: server umount lustre-OST0000 complete [16886.756044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16886.764075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16886.792450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16890.336990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16892.313979] Lustre: server umount lustre-MDT0000 complete [16895.468232] LustreError: 511466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16895.508830] LustreError: 511466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16895.946606] LustreError: 511447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565103 with bad export cookie 18389872635587601084 [16895.952677] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16895.958277] LustreError: 511447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16896.438380] Lustre: server umount lustre-MDT0001 complete [16909.976704] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [16912.658294] Key type lgssc unregistered [16912.905305] LNet: 519286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16912.917114] LNetError: 519286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16912.931335] LNet: Removed LNI 192.168.201.127@tcp [16913.797967] Key type .llcrypt unregistered [16913.799212] Key type ._llcrypt unregistered [16933.068451] Key type ._llcrypt registered [16933.070779] Key type .llcrypt registered [16933.176809] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16934.631308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16934.655929] alg: No test for adler32 (adler32-zlib) [16935.734681] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [16936.065933] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [16937.743948] Key type lgssc registered [16939.003260] Lustre: Echo OBD driver; http://www.lustre.org/ [16980.816492] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [16992.019137] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [17026.757980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17026.782299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17028.069831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17028.101283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17028.182876] Lustre: lustre-MDT0000: new disk, initializing [17028.327255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17028.357169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17033.000537] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [17045.238279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17045.398187] Lustre: 523754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17045.475157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17045.477409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [17045.478935] Lustre: Skipped 1 previous similar message [17045.563957] Lustre: lustre-MDT0001: new disk, initializing [17045.726104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17045.777158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17045.790419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17049.623522] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [17059.345118] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17065.445450] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17072.263416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17072.462896] Lustre: lustre-OST0000: new disk, initializing [17072.465628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17072.470412] Lustre: 525722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17072.541855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17078.509422] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [17081.897709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17081.907784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17082.053751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17086.853795] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17091.772906] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17092.069841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17097.684035] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17097.855885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17102.305289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17102.341572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17102.358747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17102.827238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17106.233812] Lustre: server umount lustre-OST0000 complete [17112.545586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17112.558285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17112.581492] Lustre: Skipped 1 previous similar message [17112.590238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17112.592517] Lustre: Skipped 1 previous similar message [17118.097308] Lustre: server umount lustre-MDT0000 complete [17121.154798] LustreError: 523745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565328 with bad export cookie 10716499307912095213 [17121.154798] LustreError: 524549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565328 with bad export cookie 10716499307912095213 [17121.155851] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17121.161833] LustreError: 523745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17121.473420] Lustre: server umount lustre-MDT0001 complete [17143.450753] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [17154.139218] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [17186.556754] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16834 sec ========= 05:56:33 (1787565393) [17188.132652] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:56:35 (1787565395) === [17192.183720] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:56:39 (1787565399) === [17230.064475] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [17232.956258] Key type lgssc unregistered [17233.183618] LNet: 532368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17233.190510] LNetError: 532368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17233.210416] LNet: Removed LNI 192.168.201.127@tcp [17233.990064] Key type .llcrypt unregistered [17233.995135] Key type ._llcrypt unregistered