[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479000593 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006797] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021466] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026561] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] *** VALIDATE tmpfs *** [ 0.031213] *** VALIDATE proc *** [ 0.032224] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041238] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.043891] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047024] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052384] rcu: Hierarchical SRCU implementation. [ 0.054468] smp: Bringing up secondary CPUs ... [ 0.055630] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.060009] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149020] node 0 deferred pages initialised in 83ms [ 0.153107] devtmpfs: initialized [ 0.154234] x86/mm: Memory block size: 128MB [ 0.156662] gcov: version magic: 0x41383552 [ 0.159177] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164294] pinctrl core: initialized pinctrl subsystem [ 0.165145] [ 0.167011] ************************************************************* [ 0.169012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170009] ** ** [ 0.172018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174011] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.178011] ** IOMMU data structures, which may compromise security on ** [ 0.180012] ** your system. ** [ 0.182009] ** ** [ 0.184012] ** If you see this message and you are not debugging the ** [ 0.186012] ** kernel, report this immediately to your vendor! ** [ 0.188012] ** ** [ 0.190008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191011] ************************************************************* [ 0.193591] NET: Registered protocol family 16 [ 0.195277] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202098] cpuidle: using governor menu [ 0.203996] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206612] PCI: Using configuration type 1 for base access [ 0.209134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222043] cryptd: max_cpu_qlen set to 1000 [ 0.224082] ACPI: Added _OSI(Module Device) [ 0.225011] ACPI: Added _OSI(Processor Device) [ 0.226011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230724] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238513] ACPI: Interpreter enabled [ 0.239059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241013] ACPI: Using IOAPIC for interrupt routing [ 0.242114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245482] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254955] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269348] acpiphp: Slot [2] registered [ 0.272144] acpiphp: Slot [5] registered [ 0.274143] acpiphp: Slot [6] registered [ 0.275107] acpiphp: Slot [7] registered [ 0.276125] acpiphp: Slot [8] registered [ 0.278108] acpiphp: Slot [9] registered [ 0.279147] acpiphp: Slot [10] registered [ 0.281094] acpiphp: Slot [3] registered [ 0.282114] acpiphp: Slot [4] registered [ 0.284217] acpiphp: Slot [11] registered [ 0.285099] acpiphp: Slot [12] registered [ 0.287099] acpiphp: Slot [13] registered [ 0.289109] acpiphp: Slot [14] registered [ 0.290096] acpiphp: Slot [15] registered [ 0.292104] acpiphp: Slot [16] registered [ 0.293133] acpiphp: Slot [17] registered [ 0.295136] acpiphp: Slot [18] registered [ 0.297102] acpiphp: Slot [19] registered [ 0.298091] acpiphp: Slot [20] registered [ 0.299086] acpiphp: Slot [21] registered [ 0.300102] acpiphp: Slot [22] registered [ 0.301100] acpiphp: Slot [23] registered [ 0.303107] acpiphp: Slot [24] registered [ 0.304082] acpiphp: Slot [25] registered [ 0.305083] acpiphp: Slot [26] registered [ 0.306138] acpiphp: Slot [27] registered [ 0.308091] acpiphp: Slot [28] registered [ 0.309089] acpiphp: Slot [29] registered [ 0.310168] acpiphp: Slot [30] registered [ 0.312106] acpiphp: Slot [31] registered [ 0.313102] PCI host bridge to bus 0000:00 [ 0.314020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327854] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331374] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346620] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358678] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361912] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369789] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399178] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417078] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443141] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480342] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.505019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.530017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563186] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.598019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.608621] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.616016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.623016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.660312] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.662457] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.664376] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.667363] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675027] iommu: Default domain type: Passthrough [ 0.676451] SCSI subsystem initialized [ 0.677157] ACPI: bus type USB registered [ 0.679106] usbcore: registered new interface driver usbfs [ 0.680073] usbcore: registered new interface driver hub [ 0.682069] usbcore: registered new device driver usb [ 0.683172] pps_core: LinuxPPS API ver. 1 registered [ 0.685010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.687087] PTP clock support registered [ 0.690097] EDAC MC: Ver: 3.0.0 [ 0.692151] PCI: Using ACPI for IRQ routing [ 0.694988] NetLabel: Initializing [ 0.696017] NetLabel: domain hash size = 128 [ 0.698011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699075] NetLabel: unlabeled traffic allowed by default [ 0.701115] vgaarb: loaded [ 0.703271] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.709332] clocksource: Switched to clocksource kvm-clock [ 0.816256] VFS: Disk quotas dquot_6.6.0 [ 0.818255] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821624] *** VALIDATE ramfs *** [ 0.823027] *** VALIDATE hugetlbfs *** [ 0.824859] pnp: PnP ACPI init [ 0.827363] pnp: PnP ACPI: found 6 devices [ 0.858909] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.862295] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.864512] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866669] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868670] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.870513] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.872695] NET: Registered protocol family 2 [ 0.875044] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.879481] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.883410] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890607] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894247] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897799] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900836] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904280] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907677] NET: Registered protocol family 1 [ 0.910342] RPC: Registered named UNIX socket transport module. [ 0.912661] RPC: Registered udp transport module. [ 0.914122] RPC: Registered tcp transport module. [ 0.915479] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917284] NET: Registered protocol family 44 [ 0.918588] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.920333] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.921947] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.923843] PCI: CLS 0 bytes, default 64 [ 0.925248] Unpacking initramfs... [ 2.330500] debug: unmapping init [mem 0xffffa06f7cc54000-0xffffa06f7ffbffff] [ 2.333477] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.335164] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.337684] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.838760] Initialise system trusted keyrings [ 2.840509] Key type blacklist registered [ 2.841853] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.849703] zbud: loaded [ 2.853125] *** VALIDATE nfs *** [ 2.854493] *** VALIDATE nfs4 *** [ 2.856160] pstore: using deflate compression [ 2.860466] Platform Keyring initialized [ 2.971484] NET: Registered protocol family 38 [ 2.973761] Key type asymmetric registered [ 2.976281] Asymmetric key parser 'x509' registered [ 2.978216] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.981747] io scheduler mq-deadline registered [ 2.983432] io scheduler kyber registered [ 2.985168] io scheduler bfq registered [ 2.987316] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.990686] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.993946] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997316] ACPI: Power Button [PWRF] [ 3.003310] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.011264] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.026603] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.034924] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.063219] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091081] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.118295] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.124164] Non-volatile memory driver v1.3 [ 3.125764] Linux agpgart interface v0.103 [ 3.158764] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.161386] vda: detected capacity change from 0 to 75382784 [ 3.180352] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.183197] vdb: detected capacity change from 0 to 1073741824 [ 3.197315] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.200593] vdc: detected capacity change from 0 to 2621440000 [ 3.214635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.217616] vdd: detected capacity change from 0 to 2621440000 [ 3.235270] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238591] vde: detected capacity change from 0 to 4294967296 [ 3.260111] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263244] vdf: detected capacity change from 0 to 4294967296 [ 3.274642] libphy: Fixed MDIO Bus: probed [ 3.289226] usbcore: registered new interface driver usbserial_generic [ 3.291774] usbserial: USB Serial support registered for generic [ 3.294392] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.299367] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.301641] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304159] mousedev: PS/2 mouse device common for all mice [ 3.307434] rtc_cmos 00:05: RTC can wake from S4 [ 3.310388] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.312239] rtc_cmos 00:05: registered as rtc0 [ 3.316709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.319731] intel_pstate: CPU model not supported [ 3.322821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.325721] hid: raw HID events driver (C) Jiri Kosina [ 3.329259] usbcore: registered new interface driver usbhid [ 3.331504] usbhid: USB HID core driver [ 3.333432] drop_monitor: Initializing network drop monitor service [ 3.336186] Initializing XFRM netlink socket [ 3.336205] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.338394] NET: Registered protocol family 10 [ 3.345871] Segment Routing with IPv6 [ 3.347056] NET: Registered protocol family 17 [ 3.349205] mpls_gso: MPLS GSO support [ 3.356291] RAS: Correctable Errors collector initialized. [ 3.358149] AVX version of gcm_enc/dec engaged. [ 3.359769] AES CTR mode by8 optimization enabled [ 3.441926] sched_clock: Marking stable (3441900562, 0)->(4296326279, -854425717) [ 3.446199] registered taskstats version 1 [ 3.448584] Loading compiled-in X.509 certificates [ 3.450833] zswap: loaded using pool lzo/zbud [ 3.474075] Key type big_key registered [ 3.485580] Key type encrypted registered [ 3.487090] ima: No TPM chip found, activating TPM-bypass! [ 3.489480] ima: Allocated hash algorithm: sha1 [ 3.491341] ima: No architecture policies found [ 3.493044] evm: Initialising EVM extended attributes: [ 3.495146] evm: security.selinux [ 3.496679] evm: security.ima [ 3.498332] evm: security.capability [ 3.500097] evm: HMAC attrs: 0x1 [ 3.502864] rtc_cmos 00:05: setting system clock to 2026-09-11 10:45:44 UTC (1789123544) [ 3.509990] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 3.513431] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 3.522108] Write protecting the kernel read-only data: 28672k [ 3.526111] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 3.529446] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 3.566869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.575327] systemd[1]: Detected virtualization kvm. [ 3.577551] systemd[1]: Detected architecture x86-64. [ 3.579126] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.608892] systemd[1]: No hostname configured. [ 3.610898] systemd[1]: Set hostname to . [ 3.613048] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615732] systemd[1]: Initializing machine ID from random generator. [ 3.680143] random: ln: uninitialized urandom read (6 bytes read) [ 3.788157] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793305] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.801043] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.807887] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.516370] device-mapper: uevent: version 1.0.3 [ 4.518609] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.301003] random: fast init done [ 5.309279] virtio_net virtio0 ens2: renamed from eth0 [ 5.362553] scsi host0: ata_piix [ 5.392590] scsi host1: ata_piix [ 5.394658] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.397532] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.681506] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.008343] random: crng init done [ 10.009661] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.493649] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.705760] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.056750] SELinux: Disabled at runtime. [ 12.184782] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.201541] systemd[1]: Detected virtualization kvm. [ 12.206028] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.894711] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.901472] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.924642] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.934523] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.943227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.983116] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.007212] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ 14.179479] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.447400] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.519734] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.656760] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.566460] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.685321] EDAC sbridge: Ver: 1.1.2 [ 20.697135] Key type dns_resolver registered [ 21.458591] NFS: Registering the id_resolver key type [ 21.462143] Key type id_resolver registered [ 21.464589] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 49.101061] hrtimer: interrupt took 13348301 ns [ 85.635904] libcfs: loading out-of-tree module taints kernel. [ 85.668791] Key type ._llcrypt registered [ 85.670988] Key type .llcrypt registered [ 85.745794] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 105.402519] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 107.121285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.131479] alg: No test for adler32 (adler32-zlib) [ 108.724638] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 109.742891] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 111.615206] Key type lgssc registered [ 113.613783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.947716] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.753803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 188.948717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 188.992666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.342387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.379135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.477721] Lustre: lustre-MDT0000: new disk, initializing [ 190.544957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.572641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.215854] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.524947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.599969] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.626942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.630489] Lustre: Skipped 1 previous similar message [ 210.698282] Lustre: lustre-MDT0001: new disk, initializing [ 210.761740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.790186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.846436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.091264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.414920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.594394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.818855] Lustre: lustre-OST0000: new disk, initializing [ 229.823546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.831876] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.909956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.926579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.942314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.014837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.334933] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.205911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.313643] Lustre: lustre-OST0001: new disk, initializing [ 249.318481] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.323351] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.394427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.432480] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.538272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.545219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.629834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.964745] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.276575] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.894896] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.864206] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 292.424394] Lustre: DEBUG MARKER: Client: 2.17.57.138 [ 295.067379] Lustre: DEBUG MARKER: MDS: 2.17.57.138 [ 297.404275] Lustre: DEBUG MARKER: OSS: 2.17.57.138 [ 299.065342] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 06:50:39 EDT 2026 [ 316.477671] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 317.870633] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.297250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.304124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.317567] Lustre: Skipped 3 previous similar messages [ 333.325651] Lustre: Skipped 3 previous similar messages [ 338.408988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.413023] Lustre: Skipped 3 previous similar messages [ 339.490720] Lustre: server umount lustre-MDT0000 complete [ 343.517421] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123884 with bad export cookie 1069675663504945027 [ 343.526566] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.531861] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.542135] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.556353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.640471] LustreError: 7832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.645707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.654378] LustreError: 7832:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 348.671202] Lustre: Skipped 2 previous similar messages [ 350.116317] Lustre: server umount lustre-MDT0001 complete [ 354.437970] Lustre: server umount lustre-OST0000 complete [ 358.569424] Lustre: server umount lustre-OST0001 complete [ 366.441574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 377.371564] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 414.560456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.866858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.912874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.014084] Lustre: lustre-MDT0000: new disk, initializing [ 415.133299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.171349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.787986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 432.337530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.530350] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.662927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.673566] Lustre: Skipped 1 previous similar message [ 432.804473] Lustre: lustre-MDT0001: new disk, initializing [ 432.949056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.035667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.051410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.829238] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 448.312920] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.516539] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 461.966649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.278778] Lustre: lustre-OST0000: new disk, initializing [ 462.283567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.290611] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.428484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.808453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.816724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.961792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.366877] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 479.295632] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 484.676969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.067859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.102746] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.366466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.571881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.575781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.593138] Lustre: Skipped 1 previous similar message [ 494.616383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.731149] Lustre: server umount lustre-OST0000 complete [ 504.801585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.802450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.815430] Lustre: Skipped 1 previous similar message [ 504.818176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.818184] LustreError: Skipped 1 previous similar message [ 504.858360] Lustre: Skipped 2 previous similar messages [ 511.035546] Lustre: server umount lustre-MDT0000 complete [ 514.024646] LustreError: 15089:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.036860] LustreError: 15089:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 515.224810] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124056 with bad export cookie 1069675663504946560 [ 515.246895] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 515.252092] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.588892] Lustre: server umount lustre-MDT0001 complete [ 521.094634] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:54:20 (1789124060) [ 524.343643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 526.007763] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:54:26 (1789124066) [ 533.121615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.690322] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.849420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.283722] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 539.107639] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.219085] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.038470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.698416] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 560.962402] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 566.657912] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.855967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.749429] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 589.908907] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.108342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.115285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.136582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.148816] Lustre: Skipped 1 previous similar message [ 603.175526] Lustre: Skipped 3 previous similar messages [ 607.194701] Lustre: server umount lustre-OST0000 complete [ 613.354126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.376090] Lustre: Skipped 1 previous similar message [ 618.270659] Lustre: server umount lustre-MDT0000 complete [ 618.476633] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.504968] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 622.097214] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124163 with bad export cookie 1069675663504947435 [ 622.105450] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.115075] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 622.567550] Lustre: server umount lustre-MDT0001 complete [ 636.194316] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 639.342355] Key type lgssc unregistered [ 639.660413] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.669508] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.684083] LNet: Removed LNI 192.168.204.145@tcp [ 640.711427] Key type .llcrypt unregistered [ 640.715231] Key type ._llcrypt unregistered [ 665.876343] Key type ._llcrypt registered [ 665.878646] Key type .llcrypt registered [ 666.013568] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 667.385714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.577371] alg: No test for adler32 (adler32-zlib) [ 668.713307] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 668.954594] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 670.623252] Key type lgssc registered [ 672.117239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.098959] loop: module loaded [ 680.154635] loop0: detected capacity change from 0 to 204800000 [ 680.324103] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.335261] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.536923] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.962449] loop0: detected capacity change from 0 to 204800000 [ 686.084449] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.097882] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.114401] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.973517] loop0: detected capacity change from 0 to 204800000 [ 696.362985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.386281] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.719477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 697.809864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 697.929408] Lustre: lustre-MDT0000: new disk, initializing [ 698.098579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.151546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 704.206404] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 713.771202] loop1: detected capacity change from 0 to 204800000 [ 718.715096] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.120577] Lustre: lustre-OST07c6: new disk, initializing [ 719.123381] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 719.136252] Lustre: Skipped 1 previous similar message [ 719.140911] Lustre: 26189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 719.245040] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 725.390352] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 725.669949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 725.682990] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 725.722903] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 730.416884] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 730.423759] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 730.428946] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 730.433976] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 730.441717] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 730.446665] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 748.359267] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 748.372303] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 748.380444] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.389321] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 748.398501] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 748.405834] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.476800] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 753.487127] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 753.493174] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 753.508909] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.522057] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.528216] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.533501] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 753.547735] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.560578] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 753.569568] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.575672] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.584876] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.795362] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 756.806444] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 756.818887] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 756.825153] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.833860] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 756.840376] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.852113] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 756.861363] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.871387] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.879950] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.894520] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.905127] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.617818] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 761.631305] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 761.636337] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.640661] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.653034] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.666012] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.670904] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 761.675266] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.681911] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 761.686878] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.693179] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 761.705434] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.885381] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 772.896394] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 772.901459] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.912987] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.926626] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.935310] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.943171] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 772.956143] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.962647] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.969776] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.977315] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.986223] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 822.753883] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.785435] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 826.127370] Lustre: server umount lustre-OST07c6 complete [ 835.050495] Lustre: server umount lustre-MDT0000 complete [ 851.366853] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 854.859401] Key type lgssc unregistered [ 855.267503] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.271576] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.286926] LNet: Removed LNI 192.168.204.145@tcp [ 856.130526] Key type .llcrypt unregistered [ 856.132577] Key type ._llcrypt unregistered [ 875.141667] Key type ._llcrypt registered [ 875.145317] Key type .llcrypt registered [ 875.341425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 876.467162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.577586] alg: No test for adler32 (adler32-zlib) [ 877.830202] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 878.153290] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 879.839328] Key type lgssc registered [ 881.123732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.071645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.106221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.621221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.515765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 901.091739] LustreError: 30028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.136375] LustreError: 30028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 906.213682] LustreError: 30027:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.477245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.237782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 915.448483] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 923.533894] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 928.986981] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 935.248400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 935.795794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.821731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.340366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 950.969962] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.246167] Lustre: server umount lustre-OST0000 complete [ 975.841168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 975.851336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.872979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.938533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.939944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.967737] Lustre: Skipped 1 previous similar message [ 985.060153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.073160] Lustre: Skipped 1 previous similar message [ 990.175143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 990.179197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.179627] Lustre: 29338:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124515/real 1789124515] req@00000000fbb9743c x1876032926810752/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789124531 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 990.179683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.258136] Lustre: Skipped 1 previous similar message [ 990.456102] Lustre: server umount lustre-MDT0000 complete [ 994.281260] LustreError: 30008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124535 with bad export cookie 8348586178202966080 [ 994.290987] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 994.294086] LustreError: 30008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 994.748226] Lustre: server umount lustre-MDT0001 complete [ 1006.910251] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1009.747751] Key type lgssc unregistered [ 1010.012055] LNet: 33788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.022403] LNetError: 33788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.040400] LNet: Removed LNI 192.168.204.145@tcp [ 1010.963768] Key type .llcrypt unregistered [ 1010.966593] Key type ._llcrypt unregistered [ 1030.041793] Key type ._llcrypt registered [ 1030.045461] Key type .llcrypt registered [ 1030.175734] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1031.727964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1031.790041] alg: No test for adler32 (adler32-zlib) [ 1032.915194] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1033.286410] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1034.959190] Key type lgssc registered [ 1036.021609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1043.455847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1043.483659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.145367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.695054] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1055.719585] LustreError: 35060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.733065] LustreError: 35060:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1058.919594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.302803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1064.599199] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1072.042630] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.863314] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1086.808788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.038451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.107519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1094.536540] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1103.043448] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1114.594435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.595208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.605252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.623070] Lustre: Skipped 1 previous similar message [ 1114.638970] Lustre: Skipped 1 previous similar message [ 1119.785981] Lustre: server umount lustre-OST0000 complete [ 1124.833703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.853144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.953066] LustreError: 35061:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.974594] LustreError: 35061:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1131.858442] Lustre: server umount lustre-MDT0000 complete [ 1134.059414] LustreError: 37478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.074399] LustreError: 37478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1135.821038] LustreError: 35040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124676 with bad export cookie 15994716164720863198 [ 1135.830281] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.832242] LustreError: 35040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1136.120837] Lustre: server umount lustre-MDT0001 complete [ 1147.395922] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1150.087551] Key type lgssc unregistered [ 1150.411560] LNet: 38595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.418687] LNetError: 38595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.436399] LNet: Removed LNI 192.168.204.145@tcp [ 1151.351207] Key type .llcrypt unregistered [ 1151.353714] Key type ._llcrypt unregistered [ 1173.382876] Key type ._llcrypt registered [ 1173.385077] Key type .llcrypt registered [ 1173.459581] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1174.276936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.327580] alg: No test for adler32 (adler32-zlib) [ 1175.329886] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1175.534485] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1177.175183] Key type lgssc registered [ 1178.471167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.297756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.333563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.048077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.288169] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1197.557979] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.601880] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1202.659065] LustreError: 40015:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.982897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.558485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.924985] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1215.465549] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1221.093937] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1227.054942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1227.457763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.565083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1234.045289] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1241.043580] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1247.838215] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1248.226302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.248688] Lustre: Skipped 1 previous similar message [ 1252.913776] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1252.921214] Lustre: Skipped 3 previous similar messages [ 1253.094378] Lustre: server umount lustre-MDT0000 complete [ 1253.345605] LustreError: 40744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.356386] LustreError: 40744:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1256.969544] LustreError: 42597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124797 with bad export cookie 8043608327427464507 [ 1256.971645] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.985498] LustreError: 42597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1258.030346] LustreError: 42508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.068935] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 1258.474281] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.485708] Lustre: Skipped 1 previous similar message [ 1261.473102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1261.490037] Lustre: Skipped 1 previous similar message [ 1263.167128] LustreError: 40012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.179114] LustreError: 40012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1263.426556] Lustre: server umount lustre-MDT0001 complete [ 1284.697346] Lustre: server umount lustre-OST0000 complete [ 1300.772497] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1303.483345] Key type lgssc unregistered [ 1303.746938] LNet: 43881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.756386] LNetError: 43881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.778548] LNet: Removed LNI 192.168.204.145@tcp [ 1304.903887] Key type .llcrypt unregistered [ 1304.906521] Key type ._llcrypt unregistered [ 1333.281985] Key type ._llcrypt registered [ 1333.283795] Key type .llcrypt registered [ 1333.430454] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1334.665385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.720621] alg: No test for adler32 (adler32-zlib) [ 1336.088620] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1336.507427] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1338.352590] Key type lgssc registered [ 1339.492990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.533883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.560086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.146699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.665492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1359.340961] LustreError: 45299:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.369836] LustreError: 45299:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1364.167982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.449854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1364.754257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.113850] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1378.087032] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1383.723654] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1390.730141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.075187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.158746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1397.458546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1406.264126] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.344486] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1419.747672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.751318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.763193] LustreError: Skipped 1 previous similar message [ 1419.784398] Lustre: Skipped 1 previous similar message [ 1422.455670] Lustre: server umount lustre-OST0000 complete [ 1441.762039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1441.780357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.798439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.805435] Lustre: Skipped 2 previous similar messages [ 1444.321802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.324072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.930807] Lustre: server umount lustre-MDT0000 complete [ 1449.440250] LustreError: 45300:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.467315] LustreError: 45300:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1451.161096] LustreError: 45279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124992 with bad export cookie 13532058157334891308 [ 1451.175602] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.182674] LustreError: 45279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1457.661747] Lustre: server umount lustre-MDT0001 complete [ 1469.674840] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1473.147756] Key type lgssc unregistered [ 1473.450651] LNet: 49012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.456736] LNetError: 49012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.475718] LNet: Removed LNI 192.168.204.145@tcp [ 1474.318779] Key type .llcrypt unregistered [ 1474.321542] Key type ._llcrypt unregistered [ 1499.840653] Key type ._llcrypt registered [ 1499.844884] Key type .llcrypt registered [ 1500.013155] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1501.418261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.505361] alg: No test for adler32 (adler32-zlib) [ 1502.666987] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1502.984645] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1504.751230] Key type lgssc registered [ 1506.260429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1515.232319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1515.263086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.695974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.273812] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1526.244979] LustreError: 50434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.268181] LustreError: 50434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1531.362136] LustreError: 50435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.999491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.650496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.029293] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1546.947626] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1553.028272] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.818074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.303565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.338864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1567.170825] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1575.556937] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1584.337571] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1588.113155] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1600.736597] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-11 7h12m19s [ 1602.458934] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1608.289786] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1608.676478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.700369] Lustre: Skipped 1 previous similar message [ 1609.328028] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1609.344255] Lustre: Skipped 3 previous similar messages [ 1611.025216] Lustre: server umount lustre-MDT0000 complete [ 1614.834753] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1615.406984] LustreError: 53129:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.549365] LustreError: 50431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.567934] LustreError: 50431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1622.250559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.387179] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.618903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.700347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1623.679888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1628.365434] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1632.174904] Lustre: DEBUG MARKER: Wait for df (36741) ... [ 1633.592740] Lustre: DEBUG MARKER: done [ 1637.859863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.866538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.879992] Lustre: Skipped 1 previous similar message [ 1637.884558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.892388] Lustre: Skipped 3 previous similar messages [ 1639.400402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.415544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1643.465480] Lustre: server umount lustre-OST0000 complete [ 1649.632799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.637645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1649.656968] Lustre: Skipped 1 previous similar message [ 1649.659734] Lustre: Skipped 2 previous similar messages [ 1653.916410] Lustre: server umount lustre-MDT0000 complete [ 1656.803580] LustreError: 50428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.823107] LustreError: 50428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1657.658118] LustreError: 50416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125198 with bad export cookie 7948117230210738571 [ 1657.659796] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.665776] LustreError: 50416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1658.033787] Lustre: server umount lustre-MDT0001 complete [ 1668.535405] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1671.371219] Key type lgssc unregistered [ 1671.786805] LNet: 55365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.795549] LNetError: 55365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.819123] LNet: Removed LNI 192.168.204.145@tcp [ 1672.669787] Key type .llcrypt unregistered [ 1672.673402] Key type ._llcrypt unregistered [ 1713.703133] Key type ._llcrypt registered [ 1713.705475] Key type .llcrypt registered [ 1713.826650] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1715.221777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.279915] alg: No test for adler32 (adler32-zlib) [ 1716.465566] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1716.764434] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1718.439219] Key type lgssc registered [ 1719.840228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1728.630646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1728.645937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.883536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1729.900577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1730.101176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.289222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1745.104988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.237673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1745.305653] Lustre: 57574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.680488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1751.238236] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1758.454152] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1764.313673] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1771.435669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.769416] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1771.988559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.567375] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1779.960524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1787.719290] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1795.492604] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-11 7h15m35s [ 1797.313983] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1804.701028] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1807.083132] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1807.258233] LustreError: 57583:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1827.421287] Lustre: lustre-MDT0000: Client c6f99cd1-157a-4deb-a50f-6e9559969e36 (at 192.168.204.45@tcp) reconnecting [ 1847.279371] LustreError: 57583:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1850.050125] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1853.034610] Lustre: DEBUG MARKER: done [ 1860.066779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.080032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.087033] Lustre: Skipped 1 previous similar message [ 1865.193702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1865.205058] Lustre: Skipped 1 previous similar message [ 1870.309847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.328130] Lustre: Skipped 1 previous similar message [ 1874.403608] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1874.594143] Lustre: server umount lustre-OST0000 complete [ 1879.012960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1879.030185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.043422] Lustre: Skipped 1 previous similar message [ 1879.048453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.613535] Lustre: server umount lustre-MDT0000 complete [ 1887.959844] LustreError: 58325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125428 with bad export cookie 15271359064108732936 [ 1887.961341] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.971717] LustreError: 58325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1888.236878] Lustre: server umount lustre-MDT0001 complete [ 1898.235182] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1900.920195] Key type lgssc unregistered [ 1901.194959] LNet: 61912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.200911] LNetError: 61912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.213878] LNet: Removed LNI 192.168.204.145@tcp [ 1901.885961] Key type .llcrypt unregistered [ 1901.887593] Key type ._llcrypt unregistered [ 1943.450794] Key type ._llcrypt registered [ 1943.456432] Key type .llcrypt registered [ 1943.605838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1944.580473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.602339] alg: No test for adler32 (adler32-zlib) [ 1945.682079] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1945.898233] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1947.560150] Key type lgssc registered [ 1948.974735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1952.402971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.414108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.679498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1953.714233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1953.929198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.976405] Lustre: Failing over lustre-MDT0000 [ 1955.246934] Lustre: server umount lustre-MDT0000 complete [ 1962.101889] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:18:22 (1789125502) [ 1969.037034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.422229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.842953] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1983.204617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.388472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1983.427672] Lustre: 64713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1983.671496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1988.122866] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1995.734882] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2001.199153] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2007.990822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2008.214677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2008.458457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.518668] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2016.414160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2021.534767] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2027.572696] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2034.341383] Lustre: server umount lustre-MDT0000 complete [ 2035.171726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.177308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.220846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.759733] LustreError: 65448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125578 with bad export cookie 13319724659849704482 [ 2037.760398] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2037.765867] LustreError: 65448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2037.998692] Lustre: server umount lustre-MDT0001 complete [ 2041.185568] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2045.819445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2055.142668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.722628] Lustre: 63632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125583/real 1789125583] req@0000000043866965 x1876034045646592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.773081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.816978] Lustre: Skipped 1 previous similar message [ 2062.956454] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb8d92b01d85a3d05 [ 2062.986802] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 2063.296318] LustreError: 66562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.311632] LustreError: 66562:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2063.453853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.600089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2064.422184] Lustre: 63635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125588/real 1789125588] req@00000000f1b9525c x1876034045646720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.450586] LustreError: 66563:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.883307] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2069.472761] Lustre: 63634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125593/real 1789125593] req@00000000cbdff408 x1876034045646976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.491203] LustreError: 68478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.534610] LustreError: 68478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2073.568737] Lustre: 63632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125598/real 1789125598] req@00000000df0af12a x1876034045647232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.598547] LustreError: 68477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.617158] LustreError: 68477:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2077.682599] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2077.899919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.439056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.539055] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2083.842109] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2092.616445] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.678354] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2109.409350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2109.414239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.418917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.449259] Lustre: Skipped 2 previous similar messages [ 2111.435130] Lustre: server umount lustre-MDT0000 complete [ 2114.531308] LustreError: 68479:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.551106] LustreError: 68479:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2114.845768] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125655 with bad export cookie 13319724659849706757 [ 2114.846972] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.857154] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2115.322716] Lustre: server umount lustre-MDT0001 complete [ 2122.741946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2130.827528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.007245] Lustre: 63633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125660/real 1789125660] req@000000008b6d264f x1876034045674240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.033380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.179044] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb8d92b01d85a4268 [ 2139.190636] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 2139.532249] LustreError: 66562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.611123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.667337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2144.413332] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2146.722463] Lustre: 63634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789125670/real 1789125670] req@00000000e3dec6ad x1876034045674624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789125686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.775385] Lustre: 63634:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2154.979290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.629076] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2160.711133] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2170.359625] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2178.514919] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.310172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.315145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.340842] Lustre: Skipped 2 previous similar messages [ 2190.346383] Lustre: Skipped 3 previous similar messages [ 2192.754903] Lustre: server umount lustre-MDT0000 complete [ 2195.427514] LustreError: 71169:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.446541] LustreError: 71169:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 2196.298941] LustreError: 65448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125737 with bad export cookie 13319724659849708136 [ 2196.299932] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.311590] LustreError: 65448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2196.654380] Lustre: server umount lustre-MDT0001 complete [ 2203.211253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2207.001770] Lustre: DEBUG MARKER: files compared the same [ 2219.183691] Lustre: server umount lustre-OST0000 complete [ 2235.607801] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2238.228849] Key type lgssc unregistered [ 2238.521219] LNet: 74446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.549744] LNetError: 74446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.575627] LNet: Removed LNI 192.168.204.145@tcp [ 2239.405873] Key type .llcrypt unregistered [ 2239.408281] Key type ._llcrypt unregistered [ 2261.869789] Key type ._llcrypt registered [ 2261.871839] Key type .llcrypt registered [ 2261.963490] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2262.924603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.967068] alg: No test for adler32 (adler32-zlib) [ 2264.036486] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2264.333606] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2265.984210] Key type lgssc registered [ 2267.157843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2274.657536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2274.682676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.464531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.816580] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2286.069349] LustreError: 75864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.100208] LustreError: 75864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2290.572477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.897314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2295.784369] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2304.260510] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2311.365218] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2318.223395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.525885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.577623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2324.572269] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2331.744583] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2346.465124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.473138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.474114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.585238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.595162] Lustre: Skipped 1 previous similar message [ 2356.708738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.715024] Lustre: Skipped 1 previous similar message [ 2357.216159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2357.318353] Lustre: server umount lustre-OST0000 complete [ 2361.827219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.841480] Lustre: Skipped 1 previous similar message [ 2361.850290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.861133] Lustre: Skipped 2 previous similar messages [ 2366.944318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.953122] Lustre: Skipped 1 previous similar message [ 2367.586392] Lustre: server umount lustre-MDT0000 complete [ 2371.363944] LustreError: 75844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125912 with bad export cookie 14070788445700428492 [ 2371.385359] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2371.682715] Lustre: server umount lustre-MDT0001 complete [ 2381.755393] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2384.877140] Key type lgssc unregistered [ 2385.249589] LNet: 79399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.259331] LNetError: 79399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.283605] LNet: Removed LNI 192.168.204.145@tcp [ 2386.212299] Key type .llcrypt unregistered [ 2386.215093] Key type ._llcrypt unregistered [ 2414.430600] Key type ._llcrypt registered [ 2414.433424] Key type .llcrypt registered [ 2414.608143] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2415.926694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2415.986276] alg: No test for adler32 (adler32-zlib) [ 2417.097260] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2417.302084] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2418.991216] Key type lgssc registered [ 2420.336028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2427.754069] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2427.764383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2443.231984] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2448.351329] LustreError: 80789:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 2474.976402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.253649] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2489.966956] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2498.197778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.359299] Lustre: *** cfs_fail_loc=706, val=0*** [ 2498.779403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.890779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2503.490591] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2504.177044] LustreError: 80816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.196474] LustreError: 80816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2509.300278] LustreError: 80815:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.321221] LustreError: 80815:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2512.583556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.970573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.672061] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2524.529617] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.288116] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2533.350134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2533.353733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.378377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.469728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.486361] Lustre: Skipped 1 previous similar message [ 2538.867964] Lustre: server umount lustre-OST0000 complete [ 2543.586743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.597245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.614833] Lustre: Skipped 1 previous similar message [ 2543.636410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.642803] Lustre: Skipped 2 previous similar messages [ 2548.714750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.722591] Lustre: Skipped 1 previous similar message [ 2548.914239] Lustre: server umount lustre-MDT0000 complete [ 2552.583277] LustreError: 80797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126093 with bad export cookie 2793119305444119901 [ 2552.595209] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.598710] LustreError: 80797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.088682] Lustre: server umount lustre-MDT0001 complete [ 2565.004608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2568.590417] Key type lgssc unregistered [ 2568.933852] LNet: 84408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.945471] LNetError: 84408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.971641] LNet: Removed LNI 192.168.204.145@tcp [ 2570.040415] Key type .llcrypt unregistered [ 2570.044310] Key type ._llcrypt unregistered [ 2595.980848] Key type ._llcrypt registered [ 2595.984510] Key type .llcrypt registered [ 2596.121886] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2597.220517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2597.302213] alg: No test for adler32 (adler32-zlib) [ 2598.480502] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2598.699450] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2600.423259] Key type lgssc registered [ 2601.780284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2609.933438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2609.971194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.228535] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2624.169900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.787950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2629.836783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2630.116130] LustreError: 86513:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.129649] LustreError: 86513:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2634.218456] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.243014] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2639.330279] LustreError: 86511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.343757] LustreError: 86511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2639.635714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.079434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.474698] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.488978] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2647.387693] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2649.579983] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.627701] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2657.218365] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.229776] Lustre: MGS: Client 3570aaf7-b198-4d52-964f-e446dd25d04c (at 0@lo) reconnecting [ 2657.255198] LustreError: 86527:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2657.621233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.737398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2663.220380] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2678.241763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.248427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.274902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.282258] Lustre: Skipped 1 previous similar message [ 2682.772925] Lustre: server umount lustre-OST0000 complete [ 2688.480673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.483471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.498396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2688.510525] Lustre: Skipped 2 previous similar messages [ 2688.525553] Lustre: Skipped 1 previous similar message [ 2693.296743] Lustre: server umount lustre-MDT0000 complete [ 2693.602492] LustreError: 88059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.614478] LustreError: 88059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2697.543357] Lustre: server umount lustre-MDT0001 complete [ 2702.575683] Lustre: server umount MGS complete [ 2714.696875] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:30:54 (1789126254) [ 2747.369243] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2758.336290] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2798.832430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.917465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2814.430216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.731778] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2814.840493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2814.918840] Lustre: lustre-MDT0001: new disk, initializing [ 2815.005691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2815.074310] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2815.086315] Lustre: 93687:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2815.093756] LustreError: 93687:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2820.554681] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2832.823418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.028615] Lustre: lustre-OST0000: new disk, initializing [ 2833.032317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2833.037421] Lustre: 94453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2833.095927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.402753] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2843.170060] Lustre: 94885:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000000f7ca4d x1876034730369920/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789126400 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2850.392438] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2857.285505] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.297736] Lustre: MGS: Client eea548af-d4d5-43d0-b606-7ffde66a1a16 (at 0@lo) reconnecting [ 2857.318614] LustreError: 93672:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2857.378869] Lustre: 92953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2857.418599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2857.519417] Lustre: lustre-MDT0000: new disk, initializing [ 2857.668919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.695581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2857.710681] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2858.785408] Lustre: 95521:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e19ddae0 x1876034730376448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789126415 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2862.385889] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2863.601732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2863.615526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2863.916427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2889.185587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2889.202235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.219856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.234615] Lustre: Skipped 1 previous similar message [ 2889.261178] Lustre: Skipped 1 previous similar message [ 2894.313029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.280743] Lustre: server umount lustre-OST0000 complete [ 2904.545828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2904.552208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.560677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.560684] Lustre: Skipped 1 previous similar message [ 2904.599806] Lustre: Skipped 1 previous similar message [ 2914.788624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.797794] Lustre: Skipped 4 previous similar messages [ 2915.295188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.504114] Lustre: server umount lustre-MDT0000 complete [ 2919.919720] LustreError: 93656:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.949476] LustreError: 93656:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2926.945922] Lustre: server umount lustre-MDT0001 complete [ 2931.149536] Lustre: server umount MGS complete [ 2945.027673] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:34:45 (1789126485) [ 2964.287738] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2967.003479] Key type lgssc unregistered [ 2967.272529] LNet: 98023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.279565] LNetError: 98023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.289291] LNet: Removed LNI 192.168.204.145@tcp [ 2968.163108] Key type .llcrypt unregistered [ 2968.164556] Key type ._llcrypt unregistered [ 2988.337352] Key type ._llcrypt registered [ 2988.339613] Key type .llcrypt registered [ 2988.439708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2989.945724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.004714] alg: No test for adler32 (adler32-zlib) [ 2991.183801] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2991.485110] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2993.185781] Key type lgssc registered [ 2994.703895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.949759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3003.990048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.495683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.677627] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3015.650891] LustreError: 99247:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.676880] LustreError: 99247:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3016.676411] LustreError: 99248:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.163329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.994208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.364653] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3034.340737] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3040.809608] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3049.481886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.967987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.018873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3057.588390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3066.010825] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3074.532714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.538481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.542284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3074.570307] Lustre: Skipped 1 previous similar message [ 3074.615347] Lustre: Skipped 1 previous similar message [ 3079.659488] LustreError: 101072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.680772] LustreError: 101072:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3079.800330] Lustre: server umount lustre-OST0000 complete [ 3084.770714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.773349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.800614] Lustre: Skipped 1 previous similar message [ 3089.890962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.900490] Lustre: Skipped 2 previous similar messages [ 3090.252709] Lustre: server umount lustre-MDT0000 complete [ 3093.829622] LustreError: 99230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126634 with bad export cookie 11200883040629450231 [ 3093.831641] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.840295] LustreError: 99230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3094.129126] Lustre: server umount lustre-MDT0001 complete [ 3106.052871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3109.593570] Key type lgssc unregistered [ 3109.992739] LNet: 102788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.004195] LNetError: 102788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3110.021202] LNet: Removed LNI 192.168.204.145@tcp [ 3111.156774] Key type .llcrypt unregistered [ 3111.159794] Key type ._llcrypt unregistered [ 3129.048502] Key type ._llcrypt registered [ 3129.052427] Key type .llcrypt registered [ 3129.194989] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3130.539648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3130.571975] alg: No test for adler32 (adler32-zlib) [ 3131.788449] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3132.086671] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3133.831208] Key type lgssc registered [ 3135.242799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3142.075168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3142.083311] LustreError: 103870:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3142.086964] LustreError: 103871:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3142.100445] LustreError: 103870:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3142.113146] LustreError: 103870:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3142.116672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.121315] LustreError: 103870:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3143.790968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.325741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.530854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.859965] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3164.450558] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lsmod [ 3167.338631] LustreError: 105079:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3167.338706] LustreError: 105078:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3167.356821] LustreError: 105079:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3167.366977] LustreError: 105079:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3167.378572] LustreError: 105079:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3167.385292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3167.900746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.950886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3173.347736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3173.359757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.393309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.369230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.369585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.380481] Lustre: Skipped 1 previous similar message [ 3174.407574] Lustre: Skipped 1 previous similar message [ 3177.510198] LustreError: 103905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.529203] LustreError: 103905:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3177.830721] Lustre: server umount lustre-MDT0000 complete [ 3181.765654] LustreError: 103890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126722 with bad export cookie 15635206689002683348 [ 3181.775059] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.777676] LustreError: 103890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3182.560944] LustreError: 103903:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.571190] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.617089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3182.629023] Lustre: Skipped 1 previous similar message [ 3183.711273] LustreError: 104341:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001fcc63c3 x1876035290289792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3184.032674] Lustre: server umount lustre-MDT0001 complete [ 3188.445387] Lustre: server umount lustre-OST0000 complete [ 3197.332699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.795400] LustreError: 106139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.890784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.045767] LustreError: 106140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.525283] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3207.138058] LustreError: 106139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.648064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.094072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3218.447379] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3225.701392] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3230.567211] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.696050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.072726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.238304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3243.393126] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3252.180731] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3260.993433] Lustre: server umount lustre-OST0000 complete [ 3264.480222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.483538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.491763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.186659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.197296] Lustre: Skipped 3 previous similar messages [ 3278.303171] Lustre: 103274:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126803/real 1789126803] req@0000000050e67158 x1876035290318208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789126819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3278.303619] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3278.351175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.351183] Lustre: Skipped 1 previous similar message [ 3278.685460] Lustre: server umount lustre-MDT0000 complete [ 3279.330966] Lustre: 103275:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126804/real 1789126804] req@00000000758e367b x1876035290318720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789126820 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3282.654919] LustreError: 106120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126823 with bad export cookie 15635206689002684279 [ 3282.664111] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.673237] LustreError: 106120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3283.179497] Lustre: server umount lustre-MDT0001 complete [ 3295.328779] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3298.745451] Key type lgssc unregistered [ 3299.236939] LNet: 109677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3299.258578] LNetError: 109677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3299.295333] LNet: Removed LNI 192.168.204.145@tcp [ 3300.355631] Key type .llcrypt unregistered [ 3300.359623] Key type ._llcrypt unregistered [ 3326.819983] Key type ._llcrypt registered [ 3326.822857] Key type .llcrypt registered [ 3326.952322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3327.952520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.118414] alg: No test for adler32 (adler32-zlib) [ 3329.168394] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3329.396480] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3331.094727] Key type lgssc registered [ 3332.365774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3340.443928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3340.484605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.020961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.024115] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3351.522459] LustreError: 111097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.544825] LustreError: 111097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3355.906760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.366183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3362.017304] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3369.884288] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3375.020439] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3381.024785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.284317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3386.794465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3387.373251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3392.670914] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.970298] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3400.185069] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3406.080681] Lustre: server umount lustre-OST0000 complete [ 3412.961714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.982455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3412.992493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.084115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.095210] Lustre: Skipped 2 previous similar messages [ 3422.175704] Lustre: 110409:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126947/real 1789126947] req@00000000d5996000 x1876035496869120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789126963 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3422.217611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.246899] Lustre: Skipped 1 previous similar message [ 3422.266758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.277339] Lustre: Skipped 1 previous similar message [ 3423.328380] Lustre: 110409:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126948/real 1789126948] req@000000003d5a74c8 x1876035496869760/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789126964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3424.224878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3424.385281] Lustre: server umount lustre-MDT0000 complete [ 3427.305362] LustreError: 111096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.346411] LustreError: 111096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3428.319583] Lustre: 110407:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789126953/real 1789126953] req@00000000c6bcafe3 x1876035496871424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789126969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3428.903979] LustreError: 111832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126969 with bad export cookie 48023691769063838 [ 3428.912584] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.921314] LustreError: 111832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.305383] Lustre: server umount lustre-MDT0001 complete [ 3439.639609] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3442.313916] Key type lgssc unregistered [ 3442.554994] LNet: 114747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3442.560155] LNetError: 114747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3442.577827] LNet: Removed LNI 192.168.204.145@tcp [ 3443.056800] Key type .llcrypt unregistered [ 3443.058024] Key type ._llcrypt unregistered [ 3458.685329] Key type ._llcrypt registered [ 3458.688459] Key type .llcrypt registered [ 3458.793167] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3459.759106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3459.837218] alg: No test for adler32 (adler32-zlib) [ 3460.877203] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3461.076861] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3462.719159] Key type lgssc registered [ 3464.066826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.244497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.269588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.748070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.414257] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3483.109783] LustreError: 115972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.123238] LustreError: 115972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3487.005036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.517295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3492.419686] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3499.783670] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3505.626624] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3511.977351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.355426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.241261] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3518.459037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3525.051248] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3532.769194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.774354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.782097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.793413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.800484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.145773] Lustre: server umount lustre-OST0000 complete [ 3544.033281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.034778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3544.046636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.046644] Lustre: Skipped 1 previous similar message [ 3548.497945] Lustre: server umount lustre-MDT0000 complete [ 3549.155063] LustreError: 115972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.172524] LustreError: 115972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3552.518413] LustreError: 115951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127093 with bad export cookie 17641935177791079585 [ 3552.520516] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3552.531438] LustreError: 115951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3552.874604] Lustre: server umount lustre-MDT0001 complete [ 3562.086738] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3564.812350] Key type lgssc unregistered [ 3565.027755] LNet: 119505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3565.036349] LNetError: 119505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3565.055959] LNet: Removed LNI 192.168.204.145@tcp [ 3566.035952] Key type .llcrypt unregistered [ 3566.038731] Key type ._llcrypt unregistered [ 3587.965095] Key type ._llcrypt registered [ 3587.970220] Key type .llcrypt registered [ 3588.165913] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3589.526265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3589.570824] alg: No test for adler32 (adler32-zlib) [ 3590.795624] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3591.088272] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3592.783160] Key type lgssc registered [ 3594.199785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3602.747242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3602.761303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.263428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.142884] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3614.690129] LustreError: 120922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.717800] LustreError: 120922:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3618.015498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.329802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3623.010583] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3629.275933] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3635.227351] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.322681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.829066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.992363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3651.290287] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3659.259757] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.527879] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3672.193085] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3672.202196] Lustre: Skipped 2 previous similar messages [ 3676.830530] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3676.834776] Lustre: Skipped 2 previous similar messages [ 3677.790366] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3677.847569] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3685.786899] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3685.796179] Lustre: Skipped 2 previous similar messages [ 3685.916984] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp [ 3690.116993] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.45/24@tcp in log lustre-MDT0000 [ 3690.126559] Lustre: Skipped 2 previous similar messages [ 3693.944945] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3693.952642] Lustre: Skipped 1 previous similar message [ 3701.216828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.221844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.234475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.268659] Lustre: Skipped 1 previous similar message [ 3706.359317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.372497] Lustre: Skipped 2 previous similar messages [ 3706.698378] Lustre: server umount lustre-OST0000 complete [ 3711.457797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.479982] Lustre: Skipped 1 previous similar message [ 3711.480693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.494436] Lustre: Skipped 1 previous similar message [ 3716.582374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.594323] Lustre: Skipped 1 previous similar message [ 3716.942123] Lustre: server umount lustre-MDT0000 complete [ 3720.917601] LustreError: 122764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127261 with bad export cookie 17296165288663784014 [ 3720.933556] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.720717] Lustre: server umount lustre-MDT0001 complete [ 3733.604210] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3737.157222] Key type lgssc unregistered [ 3737.484934] LNet: 125619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3737.494077] LNetError: 125619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3737.525189] LNet: Removed LNI 192.168.204.145@tcp [ 3738.559508] Key type .llcrypt unregistered [ 3738.562072] Key type ._llcrypt unregistered [ 3769.349933] Key type ._llcrypt registered [ 3769.351578] Key type .llcrypt registered [ 3769.484482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3770.897093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3770.985341] alg: No test for adler32 (adler32-zlib) [ 3772.029918] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3772.241382] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3773.920052] Key type lgssc registered [ 3775.407614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3784.836021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3784.871877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.654859] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3786.705531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.725436] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.45/24@tcp [ 3792.985272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3796.973832] LustreError: 127183:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3796.985419] LustreError: 127183:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3802.087360] LustreError: 127182:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.107507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.560630] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3803.640504] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3803.704602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.011302] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3816.537632] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3823.350840] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3830.121852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.517578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.583579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3838.349818] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3846.180733] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3859.432385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.437084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3859.451611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3859.463264] Lustre: Skipped 1 previous similar message [ 3864.546788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.561825] Lustre: Skipped 2 previous similar messages [ 3865.304746] Lustre: server umount lustre-OST0000 complete [ 3869.675475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.685937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.688600] Lustre: Skipped 1 previous similar message [ 3869.698905] Lustre: Skipped 1 previous similar message [ 3874.785671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3874.804533] Lustre: Skipped 1 previous similar message [ 3875.766800] Lustre: server umount lustre-MDT0000 complete [ 3879.714407] LustreError: 127162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127420 with bad export cookie 7483539828541835827 [ 3879.727308] LustreError: 127162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3879.734179] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.137453] Lustre: server umount lustre-MDT0001 complete [ 3892.154684] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3895.158882] Key type lgssc unregistered [ 3895.511193] LNet: 130773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3895.526086] LNetError: 130773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3895.542635] LNet: Removed LNI 192.168.204.145@tcp [ 3896.818488] Key type .llcrypt unregistered [ 3896.825076] Key type ._llcrypt unregistered [ 3924.066594] Key type ._llcrypt registered [ 3924.069288] Key type .llcrypt registered [ 3924.240147] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3941.732684] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3943.346886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3943.380273] alg: No test for adler32 (adler32-zlib) [ 3944.571541] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 3944.931783] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3946.634268] Key type lgssc registered [ 3947.877940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.057171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.114768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.409285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3985.447288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3985.562238] Lustre: lustre-MDT0000: new disk, initializing [ 3985.678987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.714488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3990.893660] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4001.510785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.634706] Lustre: 134176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4001.689928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4001.693937] Lustre: Skipped 1 previous similar message [ 4001.754977] Lustre: lustre-MDT0001: new disk, initializing [ 4001.865836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4001.924506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4001.942567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4006.789385] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4017.384573] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4024.110577] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4031.700440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4032.098506] Lustre: lustre-OST0000: new disk, initializing [ 4032.104064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4032.110201] Lustre: 136144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.219882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.482525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4039.812127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4039.830705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4039.931435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4048.853783] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4053.974145] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4054.217868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4059.613213] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4059.805632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4065.768877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.776443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.786215] LustreError: Skipped 1 previous similar message [ 4065.791586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.819773] Lustre: Skipped 1 previous similar message [ 4065.829150] Lustre: Skipped 1 previous similar message [ 4070.892260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4070.898120] Lustre: Skipped 1 previous similar message [ 4075.999618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4076.002815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.033447] Lustre: Skipped 1 previous similar message [ 4076.232030] Lustre: server umount lustre-OST0000 complete [ 4081.121476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.124958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.143397] Lustre: Skipped 1 previous similar message [ 4081.169848] Lustre: Skipped 1 previous similar message [ 4086.241189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4086.254493] Lustre: Skipped 1 previous similar message [ 4086.953189] Lustre: server umount lustre-MDT0000 complete [ 4091.102326] LustreError: 134168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127632 with bad export cookie 18240430659673010998 [ 4091.104400] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.117793] LustreError: 134168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4091.610332] Lustre: server umount lustre-MDT0001 complete [ 4100.588175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.151084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.029273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4106.212759] LustreError: 138363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.239909] LustreError: 138363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4111.329511] LustreError: 138364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.746512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.359098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4121.584081] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4130.697740] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4137.867524] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4145.806704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.157429] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4146.273109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.949209] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4162.913396] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4168.365946] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4168.686135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4174.299702] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4174.503798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4183.217550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.398727] Lustre: lustre-OST0001: new disk, initializing [ 4183.405227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4183.413099] Lustre: 141736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4183.563890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.276203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4189.284584] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4189.449509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4189.452802] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4197.991687] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4198.203823] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4203.560625] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4203.797239] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4211.115982] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a8ed8b85000.ost_server_uuid 50 [ 4213.487096] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a8ed8b85000.ost_server_uuid in IDLE state after 0 sec [ 4225.515533] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.526167] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4225.534125] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.542823] Lustre: Skipped 1 previous similar message [ 4226.650348] Lustre: server umount lustre-OST0001 complete [ 4230.682126] LustreError: 140187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.696187] LustreError: 140187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4233.187072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4233.189965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.197878] LustreError: Skipped 1 previous similar message [ 4233.222811] Lustre: Skipped 1 previous similar message [ 4235.762687] LustreError: 140187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.782300] LustreError: 140187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4237.392683] Lustre: server umount lustre-OST0000 complete [ 4244.449781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4244.466417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.225326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4256.377646] Lustre: server umount lustre-MDT0000 complete [ 4260.323383] LustreError: 138364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.341724] LustreError: 138344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127801 with bad export cookie 18240430659673011880 [ 4260.346488] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.352579] LustreError: 138364:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4260.362827] LustreError: 138344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4260.647523] Lustre: server umount lustre-MDT0001 complete [ 4281.468098] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4284.922918] Key type lgssc unregistered [ 4285.155953] LNet: 144746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4285.164780] LNetError: 144746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4285.193532] LNet: Removed LNI 192.168.204.145@tcp [ 4286.124729] Key type .llcrypt unregistered [ 4286.129470] Key type ._llcrypt unregistered [ 4333.279868] Key type ._llcrypt registered [ 4333.282714] Key type .llcrypt registered [ 4333.418564] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4351.112198] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4352.210374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4352.250413] alg: No test for adler32 (adler32-zlib) [ 4353.770407] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 4354.205579] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4355.968319] Key type lgssc registered [ 4357.446518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.455251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.506901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.794699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.816632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.916985] Lustre: lustre-MDT0000: new disk, initializing [ 4400.035534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.060906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4405.455914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4416.928864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.060541] Lustre: 148902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.186069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4417.194133] Lustre: Skipped 1 previous similar message [ 4417.280524] Lustre: lustre-MDT0001: new disk, initializing [ 4417.436356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.505730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4417.520604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.156819] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4432.117136] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4438.540826] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.878041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.325455] Lustre: lustre-OST0000: new disk, initializing [ 4447.328974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4447.337467] Lustre: 150873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4447.500099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.945281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4453.962770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4454.049774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4454.792128] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4465.073181] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.576335] Lustre: Failing over lustre-OST0000 [ 4469.729814] Lustre: server umount lustre-OST0000 complete [ 4485.087099] Lustre: 147126:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128010/real 1789128010] req@0000000036ebab5c x1876036571661184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789128026 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4485.133184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.113415] Lustre: 147127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128010/real 1789128010] req@00000000e8852a71 x1876036571661312/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789128026 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4486.137133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.202329] Lustre: 147127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128012/real 1789128012] req@0000000024955ab6 x1876036571661952/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789128028 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4487.846317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.249857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.268716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4488.307581] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4490.272095] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128015/real 1789128015] req@00000000bc10ced8 x1876036571662976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789128031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4490.295885] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4494.649178] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4495.391564] Lustre: 147126:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128020/real 1789128020] req@000000004bb5ce15 x1876036571664128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789128036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4496.505353] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4496.505603] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4498.403635] Lustre: Failing over lustre-MDT0000 [ 4498.508603] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 4498.692516] Lustre: server umount lustre-MDT0000 complete [ 4499.425356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4499.432699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.570324] LustreError: 148915:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.585504] LustreError: 148915:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4508.742483] LustreError: 150465:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.691633] LustreError: 148911:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.710772] LustreError: 148911:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4513.843311] LustreError: 148910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.831140] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789128041/real 1789128041] req@00000000d692ba85 x1876036571677056/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789128057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4516.857160] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4516.862842] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.508666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.958832] LustreError: 148912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.974951] LustreError: 148912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4526.050690] LustreError: 147124:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059a15634 x1876036571682432/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4526.458337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.550062] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4528.712206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4531.519708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4531.705119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4531.718387] Lustre: Skipped 1 previous similar message [ 4531.769748] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4531.806368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4541.922965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.936595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.955809] Lustre: Skipped 3 previous similar messages [ 4546.401837] Lustre: server umount lustre-OST0000 complete [ 4552.161616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.164951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.185286] Lustre: Skipped 1 previous similar message [ 4552.185650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4552.216125] Lustre: Skipped 2 previous similar messages [ 4556.650950] Lustre: server umount lustre-MDT0000 complete [ 4557.282163] LustreError: 148912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.295547] LustreError: 148912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4560.665791] LustreError: 148896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128101 with bad export cookie 9176004001602561161 [ 4560.679762] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.148770] Lustre: server umount lustre-MDT0001 complete [ 4573.912890] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4576.766673] Key type lgssc unregistered [ 4577.129189] LNet: 154546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4577.137496] LNetError: 154546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4577.155456] LNet: Removed LNI 192.168.204.145@tcp [ 4578.277789] Key type .llcrypt unregistered [ 4578.281215] Key type ._llcrypt unregistered [ 4615.539723] Key type ._llcrypt registered [ 4615.541717] Key type .llcrypt registered [ 4615.620889] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4616.832664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4616.856135] alg: No test for adler32 (adler32-zlib) [ 4617.932560] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 4618.210308] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4619.895961] Key type lgssc registered [ 4620.974645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4629.088660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4629.117323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.511207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.936609] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4640.740933] LustreError: 156069:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.768401] LustreError: 156069:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4644.547775] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4645.859718] LustreError: 156068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.309511] LustreError: 156097:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4648.320871] LustreError: 156097:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4648.328654] LustreError: 156097:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4648.706750] Lustre: server umount lustre-MDT0000 complete [ 4666.029899] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4677.576390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4717.435775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.700979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4717.732613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4717.843697] Lustre: lustre-MDT0000: new disk, initializing [ 4717.908359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.921094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4723.019372] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4734.534430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.683911] Lustre: 159403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4734.758051] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4734.760917] Lustre: Skipped 1 previous similar message [ 4734.808949] Lustre: lustre-MDT0001: new disk, initializing [ 4734.876503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4734.921479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4734.938223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4739.694530] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4750.090849] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4757.049456] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4765.358336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4765.718689] Lustre: lustre-OST0000: new disk, initializing [ 4765.725629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4765.731803] Lustre: 161373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4765.863379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.697977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4773.439300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4773.455024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4773.592577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4781.895140] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4786.987977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4788.945588] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:05:29 (1789128329) [ 5024.736727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.739073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.744908] Lustre: Skipped 2 previous similar messages [ 5024.748432] Lustre: Skipped 1 previous similar message [ 5029.249544] Lustre: server umount lustre-MDT0000 complete [ 5029.855932] LustreError: 159416:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.865899] LustreError: 159416:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5032.588624] LustreError: 163163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128573 with bad export cookie 9986793177442616225 [ 5032.589338] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.607238] LustreError: 163163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5034.976923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.982618] LustreError: 159411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.998043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5035.011186] Lustre: Skipped 1 previous similar message [ 5037.988277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5039.164544] Lustre: server umount lustre-MDT0001 complete [ 5049.805278] Lustre: server umount lustre-OST0000 complete [ 5058.656770] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5068.088285] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5103.811742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.081986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5104.156541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5104.280861] Lustre: lustre-MDT0000: new disk, initializing [ 5104.398288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5104.421731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5108.912206] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5120.216443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.360874] Lustre: 166032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5120.430729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5120.435798] Lustre: Skipped 1 previous similar message [ 5120.595778] Lustre: lustre-MDT0001: new disk, initializing [ 5120.715764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5120.765990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5120.771738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5125.956334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5136.100436] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5141.556107] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5148.044238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5148.307407] Lustre: lustre-OST0000: new disk, initializing [ 5148.311276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5148.318631] Lustre: 168002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5148.388172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.179733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5157.426538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5157.449914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5157.601209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5163.249668] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5168.180305] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5168.381970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5173.116821] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5173.299222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5178.338172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5178.343244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.350458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.365536] Lustre: Skipped 1 previous similar message [ 5178.375228] Lustre: Skipped 1 previous similar message [ 5181.371279] Lustre: server umount lustre-OST0000 complete [ 5187.557693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5187.566840] LustreError: Skipped 1 previous similar message [ 5187.569948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.580962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.602172] Lustre: server umount lustre-MDT0000 complete [ 5193.713072] LustreError: 166044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.734659] LustreError: 166044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5194.764136] LustreError: 168001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128735 with bad export cookie 9986793177442775055 [ 5194.775110] LustreError: 168001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5194.775346] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.127850] Lustre: server umount lustre-MDT0001 complete [ 5203.380883] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:12:23 (1789128743) [ 5218.935430] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5229.204809] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5264.391989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.740841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5264.770874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5264.848789] Lustre: lustre-MDT0000: new disk, initializing [ 5264.951253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.970808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5269.264118] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5279.749554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.879471] Lustre: 172755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5280.126968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5284.877258] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5293.646628] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5300.630816] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5309.902667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.470271] Lustre: lustre-OST0000: new disk, initializing [ 5310.474818] Lustre: Skipped 1 previous similar message [ 5310.488370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5310.497603] Lustre: Skipped 2 previous similar messages [ 5310.513382] Lustre: 174734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5319.374986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5320.277493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5320.288851] Lustre: Skipped 1 previous similar message [ 5320.299961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5320.444837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5329.163699] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5341.157895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5341.168386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.180350] Lustre: Skipped 1 previous similar message [ 5341.184425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5341.189855] Lustre: Skipped 3 previous similar messages [ 5342.916542] Lustre: server umount lustre-OST0000 complete [ 5346.800042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5346.820357] LustreError: Skipped 1 previous similar message [ 5351.397701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.411275] Lustre: Skipped 2 previous similar messages [ 5352.872992] Lustre: server umount lustre-MDT0000 complete [ 5356.521056] LustreError: 172764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.548139] LustreError: 172764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5356.557053] LustreError: 172748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128897 with bad export cookie 9986793177442775937 [ 5356.567935] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.570675] LustreError: 172748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5356.900694] Lustre: server umount lustre-MDT0001 complete [ 5365.717475] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:15:06 (1789128906) [ 5381.712968] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5391.196190] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5427.418211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.787628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5427.825750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5427.889335] Lustre: lustre-MDT0000: new disk, initializing [ 5427.970098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.974491] Lustre: Skipped 2 previous similar messages [ 5427.991071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5432.548425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5442.456645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.604401] Lustre: 179055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5442.896492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5446.993713] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5455.645533] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.650186] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5468.529942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5468.806192] Lustre: 181031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5474.703942] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5478.437247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5478.542272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5483.693093] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5493.736596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.744098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5493.757401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.771420] Lustre: Skipped 4 previous similar messages [ 5496.564170] Lustre: server umount lustre-OST0000 complete [ 5506.571469] Lustre: server umount lustre-MDT0000 complete [ 5509.089380] LustreError: 179067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.124593] LustreError: 179067:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5510.293691] LustreError: 181042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129051 with bad export cookie 9986793177442777015 [ 5510.295151] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5510.304280] LustreError: 181042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5510.613454] Lustre: server umount lustre-MDT0001 complete [ 5520.276755] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5523.306656] Key type lgssc unregistered [ 5523.617250] LNet: 182890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5523.631235] LNetError: 182890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5523.648327] LNet: Removed LNI 192.168.204.145@tcp [ 5524.702252] Key type .llcrypt unregistered [ 5524.705390] Key type ._llcrypt unregistered [ 5551.101254] Key type ._llcrypt registered [ 5551.104295] Key type .llcrypt registered [ 5551.243981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5552.188671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5552.249041] alg: No test for adler32 (adler32-zlib) [ 5553.277197] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 5553.513914] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5555.191120] Key type lgssc registered [ 5556.440215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5565.215280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5565.241595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.958610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.006371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5576.162290] LustreError: 184309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.185612] LustreError: 184309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5581.277466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.282401] LustreError: 184310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.656875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5586.290520] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5594.098160] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5600.451697] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5607.770736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.213921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.327601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5615.784793] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5623.655558] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5647.329395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.346528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.360141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.367738] Lustre: Skipped 1 previous similar message [ 5647.394527] Lustre: Skipped 1 previous similar message [ 5651.557722] Lustre: server umount lustre-OST0000 complete [ 5657.593762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.605069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5657.614029] Lustre: Skipped 1 previous similar message [ 5662.456211] Lustre: server umount lustre-MDT0000 complete [ 5662.706208] LustreError: 184309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.726353] LustreError: 184309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5666.930658] LustreError: 184290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129207 with bad export cookie 4964845576129264136 [ 5666.942043] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.420343] Lustre: server umount lustre-MDT0001 complete [ 5678.606915] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5681.943413] Key type lgssc unregistered [ 5682.272050] LNet: 187868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.285504] LNetError: 187868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.309677] LNet: Removed LNI 192.168.204.145@tcp [ 5683.447760] Key type .llcrypt unregistered [ 5683.449600] Key type ._llcrypt unregistered [ 5710.415507] Key type ._llcrypt registered [ 5710.419245] Key type .llcrypt registered [ 5710.614859] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5711.788476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5711.859103] alg: No test for adler32 (adler32-zlib) [ 5713.067636] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 5713.373677] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5715.128075] Key type lgssc registered [ 5716.744626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5725.795663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5725.841964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.346209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.364550] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5737.954516] LustreError: 189289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.989279] LustreError: 189289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5741.865623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.300917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.458097] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5755.858476] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5762.280919] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5768.936103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.214665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.280836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5775.600742] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5784.094823] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5797.471102] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 5797.857089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.858421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.863486] LustreError: Skipped 1 previous similar message [ 5797.883515] Lustre: Skipped 1 previous similar message [ 5800.899739] Lustre: server umount lustre-OST0000 complete [ 5807.936189] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a8ee013f800.ost_server_uuid 50 [ 5809.382598] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5824.480850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5824.487582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.496515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5824.500631] Lustre: Skipped 2 previous similar messages [ 5827.553399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.554214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.697898] Lustre: server umount lustre-MDT0000 complete [ 5832.687136] LustreError: 191554:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.705587] LustreError: 191554:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5833.337676] LustreError: 189268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129374 with bad export cookie 15374456542086474520 [ 5833.345458] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5833.347412] LustreError: 189268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5833.730431] Lustre: server umount lustre-MDT0001 complete [ 5843.292867] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:23:03 (1789129383) [ 5850.036554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.926558] LustreError: 193206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.028949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.231071] LustreError: 193207:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.301378] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5860.329512] LustreError: 193206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.459220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.874965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5870.101958] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5877.969741] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5883.772765] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5890.112317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5890.431931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.562521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5896.646665] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5905.674855] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.230501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.573895] Lustre: lustre-OST0001: new disk, initializing [ 5914.577663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5914.587747] Lustre: 196050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5914.792608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5921.916462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5921.926094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5922.018125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5922.219802] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5932.837990] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5936.611524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.622815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.661215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.672320] Lustre: Skipped 1 previous similar message [ 5941.755704] Lustre: server umount lustre-OST0000 complete [ 5942.758723] LustreError: 195028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.780429] LustreError: 195028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5948.090920] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5948.341169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5954.047402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5954.398371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5959.222195] LustreError: 195029:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.244465] LustreError: 195029:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5973.475886] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.478627] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.490258] LustreError: Skipped 2 previous similar messages [ 5973.498566] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5973.506806] Lustre: Skipped 2 previous similar messages [ 5973.520271] Lustre: Skipped 4 previous similar messages [ 5975.803795] Lustre: server umount lustre-OST0001 complete [ 5983.712572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5983.724654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.738721] Lustre: Skipped 1 previous similar message [ 5983.740270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.759429] Lustre: Skipped 1 previous similar message [ 5986.910558] Lustre: server umount lustre-MDT0000 complete [ 5991.173075] LustreError: 193188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129532 with bad export cookie 15374456542086475591 [ 5991.183986] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.830355] Lustre: server umount lustre-MDT0001 complete [ 6017.534944] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:25:57 (1789129557) [ 6024.572386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.914958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6024.968519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6025.159980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.961982] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6039.536562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.652691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6039.690593] Lustre: 199491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6044.185303] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6051.324947] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6056.192635] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6063.610114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.855784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6071.449067] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6073.065211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6080.664938] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.728422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6088.874672] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6096.494925] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6104.313205] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6109.153328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6109.159647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.173744] LustreError: Skipped 1 previous similar message [ 6109.175436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.201981] Lustre: Skipped 1 previous similar message [ 6113.395256] Lustre: server umount lustre-OST0000 complete [ 6115.390572] LustreError: 201342:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.419671] LustreError: 201342:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6131.175105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.079521] Lustre: server umount lustre-OST0001 complete [ 6141.409345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.416495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.429947] Lustre: Skipped 3 previous similar messages [ 6141.448316] Lustre: Skipped 7 previous similar messages [ 6154.208962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6154.477701] Lustre: server umount lustre-MDT0000 complete [ 6158.850349] LustreError: 199481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129699 with bad export cookie 15374456542086476998 [ 6158.867107] LustreError: 199481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6158.868827] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.176799] Lustre: server umount lustre-MDT0001 complete [ 6185.625266] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:28:45 (1789129725) [ 6204.522501] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6214.716591] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6254.507621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.882117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6254.935665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6255.053529] Lustre: lustre-MDT0000: new disk, initializing [ 6255.206189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.213412] Lustre: Skipped 3 previous similar messages [ 6255.231158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6261.015733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6272.825461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.948637] Lustre: 207593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6273.001773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6273.007462] Lustre: Skipped 1 previous similar message [ 6273.068342] Lustre: lustre-MDT0001: new disk, initializing [ 6273.288500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6273.310046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6280.503619] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6292.151942] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6298.047691] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6305.803713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.179598] Lustre: lustre-OST0000: new disk, initializing [ 6306.185221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6306.195212] Lustre: 209562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6312.816561] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6314.552453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6314.562158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6314.734483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6322.808580] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.721611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6330.070818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6336.659234] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6337.057384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6340.577408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.584243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.586868] LustreError: Skipped 2 previous similar messages [ 6340.604428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6340.615382] Lustre: Skipped 5 previous similar messages [ 6345.454984] Lustre: server umount lustre-OST0000 complete [ 6355.076554] Lustre: server umount lustre-MDT0000 complete [ 6355.937700] LustreError: 207600:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.965391] LustreError: 207600:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6359.302605] LustreError: 207583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129900 with bad export cookie 15374456542086478559 [ 6359.322046] LustreError: 207583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6359.324305] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6359.769411] Lustre: server umount lustre-MDT0001 complete [ 6369.149965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.160908] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6385.062277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.447604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6385.456673] Lustre: Skipped 3 previous similar messages [ 6391.262747] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6399.887046] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6406.086669] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6413.351515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.376071] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6427.259839] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.681962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6431.960967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6436.960880] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6437.414231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6440.425293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.439149] LustreError: Skipped 2 previous similar messages [ 6446.387666] Lustre: server umount lustre-OST0000 complete [ 6454.224627] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6459.403873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6488.699111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.392683] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6495.143499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.154601] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.155611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6501.120708] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6506.980770] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6515.682608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.710020] Lustre: Skipped 5 previous similar messages [ 6515.733504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6515.739637] Lustre: Skipped 6 previous similar messages [ 6517.567184] Lustre: server umount lustre-OST0000 complete [ 6527.446928] Lustre: server umount lustre-MDT0000 complete [ 6530.705283] LustreError: 213625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130071 with bad export cookie 15374456542086479441 [ 6530.709145] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.715415] LustreError: 213625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6530.993896] Lustre: server umount lustre-MDT0001 complete [ 6539.954931] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:34:39 (1789130079) [ 6546.922866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.309186] LustreError: 217745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.330233] LustreError: 217745:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6552.582421] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6562.778171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.242359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6578.780297] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6585.393092] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6594.190423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.802487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6610.117756] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6614.890180] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6618.458883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6623.926441] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6624.294930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6632.143504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.431764] Lustre: lustre-OST0001: new disk, initializing [ 6632.442561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6632.459682] Lustre: 221139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.175920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6638.182021] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6638.259760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6638.764014] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6648.914690] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6653.183122] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6653.333528] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6657.528327] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6657.820536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.163792] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6664.176599] LustreError: Skipped 3 previous similar messages [ 6666.171656] Lustre: server umount lustre-OST0001 complete [ 6671.900668] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6672.162522] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6676.903119] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6677.101462] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6706.825942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.363428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6707.372540] Lustre: Skipped 6 previous similar messages [ 6712.821084] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6712.840564] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6712.846490] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6712.861392] Lustre: Skipped 1 previous similar message [ 6712.922655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6721.019615] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.202854] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6731.356380] Lustre: server umount lustre-OST0001 complete [ 6754.022332] Lustre: server umount lustre-OST0000 complete [ 6765.138197] Lustre: server umount lustre-MDT0000 complete [ 6769.000985] LustreError: 217727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130309 with bad export cookie 15374456542086480407 [ 6769.004321] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6769.008765] LustreError: 217727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6775.463879] Lustre: server umount lustre-MDT0001 complete [ 6798.305032] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:38:58 (1789130338) [ 6804.527906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6804.857356] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6804.914134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6809.668496] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6818.327924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.513326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6818.553217] Lustre: 226385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6823.599119] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6831.184499] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6837.678223] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6844.960717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.208731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6851.784094] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6859.724941] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6870.749927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.863573] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6878.598648] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6887.020035] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.859975] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6894.163712] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6894.489583] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6899.432674] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6899.749899] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6904.894761] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a8ed1947800.ost_server_uuid 50 [ 6906.423282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a8ed1947800.ost_server_uuid in IDLE state after 0 sec [ 6907.414760] Lustre: Permanently deactivating lustre-OST0001 [ 6907.432422] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6910.411591] Lustre: Permanently reactivating lustre-OST0001 [ 6917.602090] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.606877] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6917.622184] Lustre: Skipped 12 previous similar messages [ 6917.629952] Lustre: Skipped 16 previous similar messages [ 6919.676230] Lustre: server umount lustre-OST0001 complete [ 6922.722720] LustreError: 228232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.729198] LustreError: 228232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 35 previous similar messages [ 6923.234722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.247145] LustreError: Skipped 5 previous similar messages [ 6929.402692] Lustre: server umount lustre-OST0000 complete [ 6947.295144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6947.483655] Lustre: server umount lustre-MDT0000 complete [ 6950.918665] LustreError: 226378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130491 with bad export cookie 15374456542086481695 [ 6950.923496] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.931800] LustreError: 226378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6951.134274] Lustre: server umount lustre-MDT0001 complete [ 6974.417288] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:41:54 (1789130514) [ 6981.475953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.849570] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6981.898675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6981.914318] Lustre: Skipped 5 previous similar messages [ 6986.421619] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6995.329592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.427491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6995.459372] Lustre: 233079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6999.981272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7006.152447] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7011.431666] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7018.870598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.064870] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7019.095804] Lustre: Permanently deactivating lustre-OST0000 [ 7019.121953] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7025.229902] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7028.241693] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7028.256071] LustreError: 235439:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 7028.323974] LustreError: 188595:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c8d578c9 x1876037997113984/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7032.032852] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7039.860305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.930620] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7046.366575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7047.283132] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7055.216135] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7056.509918] Lustre: Permanently reactivating lustre-OST0000 [ 7056.522788] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7056.527430] Lustre: Skipped 2 previous similar messages [ 7066.221914] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7066.239848] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7066.259170] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7066.351389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7077.473204] Lustre: server umount lustre-OST0001 complete [ 7087.468876] Lustre: server umount lustre-OST0000 complete [ 7097.853271] Lustre: server umount lustre-MDT0000 complete [ 7101.429595] LustreError: 235958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130642 with bad export cookie 15374456542086483340 [ 7101.430911] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.443027] LustreError: 235958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7101.786802] Lustre: server umount lustre-MDT0001 complete [ 7113.090366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7115.964677] Key type lgssc unregistered [ 7116.269463] LNet: 237962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7116.281036] LNetError: 237962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7116.302938] LNet: Removed LNI 192.168.204.145@tcp [ 7117.122205] Key type .llcrypt unregistered [ 7117.123978] Key type ._llcrypt unregistered [ 7139.124766] Key type ._llcrypt registered [ 7139.128160] Key type .llcrypt registered [ 7139.265877] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7140.400811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7140.469045] alg: No test for adler32 (adler32-zlib) [ 7141.559477] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7141.808538] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7143.527127] Key type lgssc registered [ 7144.633612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7155.148213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7155.186989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.788159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.332073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7166.951510] LustreError: 239435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.973448] LustreError: 239435:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7169.905825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.034481] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7170.071765] Lustre: 239424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7170.078987] Lustre: Permanently deactivating lustre-MDT0001 [ 7170.090049] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7170.168647] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7170.178041] LustreError: 239465:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7170.200190] LustreError: 239465:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7170.218979] LustreError: 239465:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7170.413496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.173767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7182.732747] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7187.900734] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7194.803606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7195.262208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.271781] LustreError: 241281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.388401] LustreError: 241280:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.401980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7201.403129] LustreError: 241280:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7201.603557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7206.882338] LustreError: 241280:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.904431] LustreError: 241280:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7208.182750] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7212.003977] LustreError: 241282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.026970] LustreError: 241282:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7214.886265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.165461] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.243185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7220.248326] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7221.259097] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7228.292616] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.445996] Lustre: Permanently reactivating lustre-MDT0001 [ 7232.463721] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7232.470144] Lustre: Skipped 2 previous similar messages [ 7237.709833] Lustre: lustre-MDT0001: Client fe709571-b482-4085-a7d5-8ba76c9a221a (at 192.168.204.45@tcp) reconnecting [ 7243.522566] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7245.291605] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7251.199661] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7251.379822] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7255.732954] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7256.005974] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7258.639474] Lustre: Permanently deactivating lustre-MDT0001 [ 7258.664840] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7258.672563] Lustre: Skipped 2 previous similar messages [ 7264.421279] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7275.727477] Lustre: server umount lustre-MDT0000 complete [ 7276.518118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.524229] LustreError: 239431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.536323] Lustre: Skipped 2 previous similar messages [ 7276.561305] LustreError: 239431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7279.907335] LustreError: 239416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130820 with bad export cookie 1431535024061280598 [ 7279.908798] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.926854] LustreError: 239416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7280.350438] Lustre: server umount lustre-MDT0001 complete [ 7286.234513] Lustre: server umount lustre-OST0000 complete [ 7298.172151] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:47:18 (1789130838) [ 7310.138256] Lustre: server umount lustre-OST0001 complete [ 7317.973270] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7328.189344] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7366.288616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.533558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7366.543308] Lustre: Skipped 2 previous similar messages [ 7366.580784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7366.702790] Lustre: lustre-MDT0000: new disk, initializing [ 7366.762313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.776729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7371.678475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7382.760150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.876399] Lustre: 247693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7382.964271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7382.970205] Lustre: Skipped 1 previous similar message [ 7383.035788] Lustre: lustre-MDT0001: new disk, initializing [ 7383.138574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.214767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7383.221658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7388.281084] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7397.629811] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7404.329683] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7411.006209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.251413] Lustre: lustre-OST0000: new disk, initializing [ 7411.254156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7411.259768] Lustre: 249661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7411.338350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.594460] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7421.538481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7421.550206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7421.702792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7426.450870] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7429.346194] LustreError: 247697:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7438.617199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.794151] Lustre: lustre-OST0001: new disk, initializing [ 7438.797940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7438.804134] Lustre: 250776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7439.392540] LustreError: 247697:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7439.429487] LustreError: 247699:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7445.651960] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7448.153591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7448.160916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7448.236755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7449.464071] LustreError: 247699:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7454.630554] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7456.101889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7458.786190] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.792979] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.794063] LustreError: Skipped 1 previous similar message [ 7458.799143] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7458.808655] Lustre: Skipped 2 previous similar messages [ 7458.845366] Lustre: Skipped 1 previous similar message [ 7462.488853] Lustre: lustre-OST0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 7463.906460] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7464.361216] Lustre: server umount lustre-OST0001 complete [ 7467.567260] LustreError: 249642:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.361390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7483.373188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.395539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7483.398209] Lustre: Skipped 1 previous similar message [ 7487.468300] Lustre: server umount lustre-OST0000 complete [ 7494.625113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.627352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.650617] Lustre: Skipped 2 previous similar messages [ 7494.670757] Lustre: Skipped 3 previous similar messages [ 7504.867720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7504.882340] Lustre: Skipped 3 previous similar messages [ 7505.375080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7505.619604] Lustre: server umount lustre-MDT0000 complete [ 7509.179486] LustreError: 250775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131050 with bad export cookie 1431535024061283195 [ 7509.183302] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.191555] LustreError: 250775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7515.633118] Lustre: server umount lustre-MDT0001 complete [ 7527.013720] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7530.084676] Key type lgssc unregistered [ 7530.428354] LNet: 252813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7530.453771] LNetError: 252813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7530.475942] LNet: Removed LNI 192.168.204.145@tcp [ 7531.487235] Key type .llcrypt unregistered [ 7531.488835] Key type ._llcrypt unregistered [ 7569.687406] Key type ._llcrypt registered [ 7569.690492] Key type .llcrypt registered [ 7569.787985] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7570.711323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7570.776080] alg: No test for adler32 (adler32-zlib) [ 7571.846347] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7572.096639] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7573.759852] Key type lgssc registered [ 7575.113786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.058685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.108393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.391798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7584.438190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7584.600386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.699264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7597.994076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.095872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7598.155403] Lustre: 255021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7598.508569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7603.267123] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7611.264610] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7617.100623] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7623.681891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7623.981442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7624.193729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.613427] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7632.596178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7639.770130] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7694.307114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7694.314044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.317108] LustreError: Skipped 1 previous similar message [ 7694.330725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.331480] Lustre: Skipped 1 previous similar message [ 7694.353099] Lustre: Skipped 1 previous similar message [ 7699.128907] Lustre: server umount lustre-OST0000 complete [ 7703.817288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7716.936528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.298709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.486628] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7731.232963] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7734.704376] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7734.721647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7734.733791] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7734.743383] Lustre: Skipped 1 previous similar message [ 7770.592995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.596368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7770.615686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7770.622978] Lustre: Skipped 1 previous similar message [ 7775.713361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7775.722272] Lustre: Skipped 1 previous similar message [ 7776.480489] Lustre: server umount lustre-OST0000 complete [ 7780.834671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.851996] Lustre: Skipped 1 previous similar message [ 7780.858288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7780.865913] Lustre: Skipped 1 previous similar message [ 7785.953189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7785.961955] Lustre: Skipped 1 previous similar message [ 7786.773793] Lustre: server umount lustre-MDT0000 complete [ 7790.490160] LustreError: 255014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131331 with bad export cookie 10495835211095055539 [ 7790.492679] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7790.505862] LustreError: 255014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7790.806269] Lustre: server umount lustre-MDT0001 complete [ 7802.606328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7805.350560] Key type lgssc unregistered [ 7805.659411] LNet: 260447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7805.671618] LNetError: 260447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7805.702773] LNet: Removed LNI 192.168.204.145@tcp [ 7806.619650] Key type .llcrypt unregistered [ 7806.621408] Key type ._llcrypt unregistered [ 7830.228179] Key type ._llcrypt registered [ 7830.230216] Key type .llcrypt registered [ 7830.413522] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7831.620982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7831.687521] alg: No test for adler32 (adler32-zlib) [ 7832.888282] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7833.163433] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7834.863328] Key type lgssc registered [ 7836.249828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7845.293972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7845.325045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.927409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.044629] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7856.098047] LustreError: 261864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.140301] LustreError: 261864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7861.216973] LustreError: 261865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.742532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.288463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7867.259328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7873.934412] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7879.734330] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7887.716958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.185410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.243380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7896.116465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7904.244804] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7932.384705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7932.398683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.410096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.408228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.410514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.419898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.452286] Lustre: Skipped 1 previous similar message [ 7935.221690] Lustre: server umount lustre-OST0000 complete [ 7943.650685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.659690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7943.666552] Lustre: Skipped 1 previous similar message [ 7943.682216] Lustre: Skipped 1 previous similar message [ 7946.093611] Lustre: server umount lustre-MDT0000 complete [ 7948.769159] LustreError: 261864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.784485] LustreError: 261864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7950.078086] LustreError: 261845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131491 with bad export cookie 10943082934618818280 [ 7950.084599] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7950.097102] LustreError: 261845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7950.650551] Lustre: server umount lustre-MDT0001 complete [ 7962.041259] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7965.221817] Key type lgssc unregistered [ 7965.562490] LNet: 266174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7965.568916] LNetError: 266174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7965.588075] LNet: Removed LNI 192.168.204.145@tcp [ 7966.599099] Key type .llcrypt unregistered [ 7966.602298] Key type ._llcrypt unregistered [ 7986.644075] Key type ._llcrypt registered [ 7986.648092] Key type .llcrypt registered [ 7986.849246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7988.105594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7988.294033] alg: No test for adler32 (adler32-zlib) [ 7989.448909] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7989.723083] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7991.447472] Key type lgssc registered [ 7992.659875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8001.615670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8001.635736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.277181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.168081] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8013.798481] LustreError: 267396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.831188] LustreError: 267396:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8017.344848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.753035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8023.299602] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8030.692436] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8035.846619] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8041.962800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.324442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.489176] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8048.520365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8055.270753] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8069.100877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.114317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8069.115791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8069.120466] Lustre: Skipped 1 previous similar message [ 8069.149446] LustreError: Skipped 1 previous similar message [ 8072.123225] Lustre: server umount lustre-OST0000 complete [ 8079.330155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.333437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8079.334271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8079.334276] Lustre: Skipped 1 previous similar message [ 8079.347047] Lustre: Skipped 1 previous similar message [ 8084.448134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.454274] Lustre: Skipped 2 previous similar messages [ 8089.571178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.588609] Lustre: Skipped 1 previous similar message [ 8090.079150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8090.429601] Lustre: server umount lustre-MDT0000 complete [ 8093.862033] LustreError: 267377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131634 with bad export cookie 14461880011425069304 [ 8093.863278] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.884259] LustreError: 267377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8094.373897] Lustre: server umount lustre-MDT0001 complete [ 8104.451391] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8107.019368] Key type lgssc unregistered [ 8107.362300] LNet: 271079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8107.372372] LNetError: 271079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8108.393261] LNet: Removed LNI 192.168.204.145@tcp [ 8109.105132] Key type .llcrypt unregistered [ 8109.110692] Key type ._llcrypt unregistered [ 8132.720894] Key type ._llcrypt registered [ 8132.724503] Key type .llcrypt registered [ 8132.858746] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8133.698546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8133.776304] alg: No test for adler32 (adler32-zlib) [ 8134.935546] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8135.204946] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8136.927651] Key type lgssc registered [ 8138.177135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8144.794332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8144.812279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.186491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.125190] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8156.642543] LustreError: 272508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.666528] LustreError: 272508:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8159.922313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.545194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8165.768960] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8174.470510] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8180.170338] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8187.059276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.424458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.565775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8194.112170] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8201.931921] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8231.397665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8231.424967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8231.438366] Lustre: Skipped 1 previous similar message [ 8236.080840] Lustre: server umount lustre-OST0000 complete [ 8241.635446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.652134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8241.661016] Lustre: Skipped 2 previous similar messages [ 8241.677137] Lustre: Skipped 1 previous similar message [ 8246.547267] Lustre: server umount lustre-MDT0000 complete [ 8246.752749] LustreError: 272508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.766922] LustreError: 272508:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8250.206207] LustreError: 272490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131791 with bad export cookie 18097330301685393294 [ 8250.215381] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8250.221209] LustreError: 272490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8250.588708] Lustre: server umount lustre-MDT0001 complete [ 8260.289352] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8263.754943] Key type lgssc unregistered [ 8264.184121] LNet: 276821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.189214] LNetError: 276821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.213953] LNet: Removed LNI 192.168.204.145@tcp [ 8265.211950] Key type .llcrypt unregistered [ 8265.213528] Key type ._llcrypt unregistered [ 8284.761565] Key type ._llcrypt registered [ 8284.764978] Key type .llcrypt registered [ 8284.977386] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8287.575677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8287.704137] alg: No test for adler32 (adler32-zlib) [ 8288.797321] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8289.349288] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8291.199176] Key type lgssc registered [ 8293.041344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8301.285356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8301.312758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.323778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8309.525263] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8313.844500] LustreError: 278057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8313.887276] LustreError: 278057:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8318.946107] LustreError: 278059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8321.990982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.086764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8328.669425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8337.785874] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8343.636095] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.262990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.699791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.880232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8363.794855] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8373.243054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8387.554387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8387.562836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8387.578768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8388.578245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8388.591030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8388.598563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.621878] Lustre: Skipped 1 previous similar message [ 8391.190949] Lustre: server umount lustre-OST0000 complete [ 8394.720596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8394.735253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.756690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.817436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8398.840641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.857903] Lustre: Skipped 1 previous similar message [ 8400.991696] Lustre: server umount lustre-MDT0000 complete [ 8403.939716] LustreError: 278061:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.963427] LustreError: 278061:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8404.510664] LustreError: 278022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131945 with bad export cookie 7891779109913780041 [ 8404.512586] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8404.527062] LustreError: 278022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8404.889475] Lustre: server umount lustre-MDT0001 complete [ 8415.817981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8418.605107] Key type lgssc unregistered [ 8418.899524] LNet: 281784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8418.909790] LNetError: 281784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.930922] LNet: Removed LNI 192.168.204.145@tcp [ 8419.691658] Key type .llcrypt unregistered [ 8419.694519] Key type ._llcrypt unregistered [ 8431.826923] Key type ._llcrypt registered [ 8431.830272] Key type .llcrypt registered [ 8431.948745] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8450.257621] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8466.122841] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8467.358706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8467.399227] alg: No test for adler32 (adler32-zlib) [ 8468.517410] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8468.789865] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8470.519265] Key type lgssc registered [ 8471.670920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8506.380107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8506.410140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.683904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8507.713869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8507.876945] Lustre: lustre-MDT0000: new disk, initializing [ 8507.992126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8508.011254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8512.445452] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8521.975598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.141368] Lustre: 285339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8522.187142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8522.189420] Lustre: Skipped 1 previous similar message [ 8522.238597] Lustre: lustre-MDT0001: new disk, initializing [ 8522.349856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8522.393404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8522.413207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8527.186371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8537.553556] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8543.124924] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8549.868424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8550.086927] Lustre: lustre-OST0000: new disk, initializing [ 8550.090374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8550.095209] Lustre: 287307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8550.183729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.479706] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8557.671682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8557.683350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8557.798632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8565.167446] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8569.782493] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8569.934117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.519405] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8573.772316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8578.536065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8578.540996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8578.556793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8578.573737] Lustre: Skipped 1 previous similar message [ 8581.644132] Lustre: server umount lustre-OST0000 complete [ 8588.773196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8588.778849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8588.785076] Lustre: Skipped 2 previous similar messages [ 8591.781309] Lustre: server umount lustre-MDT0000 complete [ 8595.963767] LustreError: 285330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132136 with bad export cookie 11366050340139884117 [ 8595.971713] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8595.978946] LustreError: 285330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8596.383943] Lustre: server umount lustre-MDT0001 complete [ 8605.012123] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:09:05 (1789132145) [ 8612.117651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.520972] LustreError: 289717:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.541875] LustreError: 289717:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8612.603482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.592482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8617.963783] LustreError: 289718:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8622.054283] LustreError: 289717:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.621787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.054798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.229612] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8640.667467] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8646.981796] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8655.378547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.804833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.539418] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8671.414887] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8681.449185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.451376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8681.467250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8681.480032] Lustre: Skipped 1 previous similar message [ 8686.578389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8686.582470] Lustre: Skipped 1 previous similar message [ 8686.758181] Lustre: server umount lustre-OST0000 complete [ 8691.689068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.706042] Lustre: Skipped 1 previous similar message [ 8691.710515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8697.324775] Lustre: server umount lustre-MDT0000 complete [ 8700.946550] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132241 with bad export cookie 11366050340139884999 [ 8700.950685] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.965070] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8701.279344] Lustre: server umount lustre-MDT0001 complete [ 8712.996769] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8715.887280] Key type lgssc unregistered [ 8716.234820] LNet: 293260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8716.244487] LNetError: 293260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8716.279850] LNet: Removed LNI 192.168.204.145@tcp [ 8717.568609] Key type .llcrypt unregistered [ 8717.570928] Key type ._llcrypt unregistered [ 8745.612339] Key type ._llcrypt registered [ 8745.617696] Key type .llcrypt registered [ 8745.764952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8763.307100] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8764.583271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8764.688326] alg: No test for adler32 (adler32-zlib) [ 8765.898156] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8766.212515] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8767.930713] Key type lgssc registered [ 8768.694767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8814.766064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8814.780979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.025487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8816.055751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8816.186740] Lustre: lustre-MDT0000: new disk, initializing [ 8816.294521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.330717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8820.280816] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8830.184225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.294032] Lustre: 297090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8830.339828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8830.343646] Lustre: Skipped 1 previous similar message [ 8830.418666] Lustre: lustre-MDT0001: new disk, initializing [ 8830.467187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8830.506919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8830.516528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8834.301320] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8843.223964] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8848.207623] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8854.802468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.061634] Lustre: lustre-OST2710: new disk, initializing [ 8855.065144] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8855.077734] Lustre: 299058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8855.189455] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8861.799718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8861.832029] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8861.966287] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8862.155342] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8872.484785] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8879.590731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8879.778749] Lustre: lustre-OST03e8: new disk, initializing [ 8879.782720] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8879.788303] Lustre: 300123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8879.918884] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8885.580452] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8887.401370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8887.414781] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8887.529453] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8893.113375] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8900.473815] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8900.766914] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8905.444501] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8905.660874] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8909.752627] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8909.998506] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8914.648728] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8914.867665] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8921.672656] Lustre: server umount lustre-MDT0000 complete [ 8922.600692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8922.616086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.635791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.662925] Lustre: Skipped 2 previous similar messages [ 8925.631201] LustreError: 297895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132466 with bad export cookie 17974768922070205542 [ 8925.640643] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.646176] LustreError: 297895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8926.130918] Lustre: server umount lustre-MDT0001 complete [ 8939.267792] Lustre: server umount lustre-OST2710 complete [ 8953.749569] Lustre: server umount lustre-OST03e8 complete [ 8973.967819] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8985.394906] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9019.443928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.697534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9019.733141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9019.799355] Lustre: lustre-MDT0000: new disk, initializing [ 9019.863684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.877868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9024.080899] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9034.234534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.348978] Lustre: 305370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9034.390456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9034.393131] Lustre: Skipped 1 previous similar message [ 9034.454618] Lustre: lustre-MDT0001: new disk, initializing [ 9034.570084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.646101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9034.660210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9038.747438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9047.687527] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9052.468129] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9058.811121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.470420] Lustre: lustre-OST0000: new disk, initializing [ 9059.487545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9059.500768] Lustre: 307340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9059.638039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.972632] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9067.066087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9067.075034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9067.158370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9074.258842] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9079.012350] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9079.210818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9084.376232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9084.724963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9087.970798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.976496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.979474] LustreError: Skipped 1 previous similar message [ 9087.989565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9088.004366] Lustre: Skipped 1 previous similar message [ 9092.796534] Lustre: server umount lustre-OST0000 complete [ 9098.211121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.220744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.232226] Lustre: Skipped 2 previous similar messages [ 9098.245571] Lustre: Skipped 1 previous similar message [ 9103.056563] Lustre: server umount lustre-MDT0000 complete [ 9103.328046] LustreError: 306291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.345293] LustreError: 306291:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9106.690078] LustreError: 305365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132647 with bad export cookie 17974768922070207229 [ 9106.707510] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9106.710725] LustreError: 305365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9107.166603] Lustre: server umount lustre-MDT0001 complete [ 9115.973254] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:17:36 (1789132656) [ 9117.627421] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9119.563972] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:17:39 (1789132659) [ 9123.474799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9152.895429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.252402] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9153.282801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9153.502510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.218563] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9166.604415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.682595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9166.700096] Lustre: 311197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9172.041795] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9193.262849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.455557] Lustre: Denying initial registration attempt from nid 192.168.204.145@tcp, specified as failover [ 9193.460748] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9193.466472] LustreError: 312652:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9193.471644] LustreError: 312652:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9193.475333] LustreError: 312652:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9193.530712] Lustre: server umount lustre-OST0000 complete [ 9193.534301] LustreError: 312652:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9200.097416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.102787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9200.120186] Lustre: Skipped 1 previous similar message [ 9200.129810] Lustre: Skipped 1 previous similar message [ 9205.217024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9205.223307] Lustre: Skipped 1 previous similar message [ 9205.759248] Lustre: server umount lustre-MDT0000 complete [ 9209.165923] LustreError: 311192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132750 with bad export cookie 17974768922070208111 [ 9209.173972] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9209.350779] Lustre: server umount lustre-MDT0001 complete [ 9221.296869] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9224.071823] Key type lgssc unregistered [ 9224.422833] LNet: 313775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9224.433576] LNetError: 313775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9224.480551] LNet: Removed LNI 192.168.204.145@tcp [ 9225.459892] Key type .llcrypt unregistered [ 9225.467716] Key type ._llcrypt unregistered [ 9235.833156] Key type ._llcrypt registered [ 9235.835020] Key type .llcrypt registered [ 9235.937727] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9236.926278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9236.969270] alg: No test for adler32 (adler32-zlib) [ 9237.974148] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 9238.116505] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9239.759559] Key type lgssc registered [ 9240.842585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9289.679887] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9298.252557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9298.273348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.496426] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9299.559455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9299.659109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.409807] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9311.979434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9312.130475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9312.187810] Lustre: 316543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9312.572809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9318.423453] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9328.093776] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9333.977166] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9353.391408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9353.582081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9353.786986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.047387] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9366.391034] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9368.545913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9368.552887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9368.571322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.072415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9369.083661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.101079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.107146] Lustre: Skipped 1 previous similar message [ 9370.513519] Lustre: server umount lustre-OST0000 complete [ 9374.177669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.185637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9374.209747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9374.221349] Lustre: Skipped 1 previous similar message [ 9379.297556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9379.308975] Lustre: Skipped 1 previous similar message [ 9380.473117] Lustre: server umount lustre-MDT0000 complete [ 9384.123751] LustreError: 316534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132925 with bad export cookie 13325060919773837037 [ 9384.129223] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9384.138510] LustreError: 316534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9384.492788] Lustre: server umount lustre-MDT0001 complete [ 9394.454511] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9397.086372] Key type lgssc unregistered [ 9397.393285] LNet: 320117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9397.412538] LNetError: 320117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9397.435918] LNet: Removed LNI 192.168.204.145@tcp [ 9398.177253] Key type .llcrypt unregistered [ 9398.179549] Key type ._llcrypt unregistered [ 9420.808356] Key type ._llcrypt registered [ 9420.811492] Key type .llcrypt registered [ 9420.892506] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9421.884671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9422.055942] alg: No test for adler32 (adler32-zlib) [ 9423.121906] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 9423.306814] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9424.975225] Key type lgssc registered [ 9426.499314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9435.431656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9435.448884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.925054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.238456] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9446.378836] LustreError: 321528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.401677] LustreError: 321528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9450.117368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.562908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9454.863953] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9462.460336] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9467.870684] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9475.689988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.949447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9480.636184] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9487.192602] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.547649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9496.550948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.557373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9496.567489] Lustre: Skipped 1 previous similar message [ 9499.205805] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 9499.212794] Lustre: Skipped 2 previous similar messages [ 9501.538553] Lustre: server umount lustre-MDT0000 complete [ 9501.665524] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.687787] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9504.306326] LustreError: 321524:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9505.146882] LustreError: 321509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133046 with bad export cookie 13748061208175071670 [ 9505.148611] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9505.158858] LustreError: 321509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9506.785973] LustreError: 323440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.788305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.808579] Lustre: Skipped 1 previous similar message [ 9506.812306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9509.468354] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 9510.817863] LustreError: 322768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.840150] LustreError: 322768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9511.481564] Lustre: server umount lustre-MDT0001 complete [ 9517.339996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9526.582987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.339851] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbecaecd4179a6d9a [ 9530.353042] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 9530.623103] LustreError: 323360:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9530.706544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9530.777857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9535.115944] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9540.969576] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9544.271436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.878136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9550.078557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9550.320676] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9561.525546] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9564.062527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9571.137721] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9573.195491] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9580.513062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9580.520619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.537365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9580.545366] Lustre: Skipped 1 previous similar message [ 9591.267372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9591.280282] Lustre: Skipped 4 previous similar messages [ 9593.311509] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9593.624825] Lustre: server umount lustre-OST0000 complete [ 9600.992856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9601.004161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.015192] Lustre: Skipped 1 previous similar message [ 9603.631539] Lustre: server umount lustre-MDT0000 complete [ 9606.626540] LustreError: 325124:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9606.659740] LustreError: 325124:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9607.732995] LustreError: 322261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133148 with bad export cookie 13748061208175078810 [ 9607.736476] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9607.743469] LustreError: 322261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9608.084948] Lustre: server umount lustre-MDT0001 complete [ 9621.942116] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9625.985725] Key type lgssc unregistered [ 9626.386792] LNet: 327800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9626.396608] LNetError: 327800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9626.422031] LNet: Removed LNI 192.168.204.145@tcp [ 9627.487831] Key type .llcrypt unregistered [ 9627.489996] Key type ._llcrypt unregistered [ 9656.711663] Key type ._llcrypt registered [ 9656.714085] Key type .llcrypt registered [ 9656.804310] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9658.402118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9658.419828] alg: No test for adler32 (adler32-zlib) [ 9659.585833] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 9659.919051] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9661.727187] Key type lgssc registered [ 9663.186743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9672.035731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9672.050438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9673.434710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.142367] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9683.940708] LustreError: 329218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.966451] LustreError: 329218:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9685.912061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.162471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9690.826881] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9698.620389] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9704.880053] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9709.028202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.043189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.053982] Lustre: Skipped 1 previous similar message [ 9711.078163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.089487] Lustre: Skipped 1 previous similar message [ 9714.581780] Lustre: server umount lustre-MDT0000 complete [ 9716.194776] LustreError: 329213:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.222884] LustreError: 329213:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9718.476883] LustreError: 329200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133259 with bad export cookie 5369659223429037647 [ 9718.480761] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.493892] LustreError: 329200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9718.907731] Lustre: server umount lustre-MDT0001 complete [ 9735.931669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.367645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9736.454413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9736.828470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.859819] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9751.476457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.592568] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9751.623290] Lustre: 331808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9751.944354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9757.144222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9765.214774] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9770.926395] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9781.168081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.341090] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9781.348666] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9781.356641] LustreError: 333692:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9781.361517] LustreError: 333692:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9781.365639] LustreError: 333692:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9781.435820] Lustre: server umount lustre-OST0000 complete [ 9781.437833] LustreError: 333692:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9792.933337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9793.142733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9793.355142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.389431] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9800.830155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9806.604401] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9814.352748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9814.487951] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9814.554494] Lustre: lustre-OST0001: new disk, initializing [ 9814.557616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9814.562977] Lustre: 335232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9814.673369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9821.811736] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9822.850911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9822.857983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9822.968211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9828.853742] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9833.453901] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9833.456891] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9833.459848] LustreError: Skipped 1 previous similar message [ 9833.461558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9833.497130] Lustre: Skipped 2 previous similar messages [ 9838.567623] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9838.576769] Lustre: Skipped 2 previous similar messages [ 9843.681882] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9843.697850] Lustre: Skipped 1 previous similar message [ 9846.239331] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9846.385514] Lustre: server umount lustre-OST0001 complete [ 9848.840557] LustreError: 334160:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.890337] LustreError: 334160:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9851.873231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9851.879743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9851.896254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9853.934571] LustreError: 334159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.967733] LustreError: 334159:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9859.041703] LustreError: 335863:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.066401] LustreError: 335863:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9865.185977] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9865.306440] Lustre: server umount lustre-OST0000 complete [ 9869.281754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9869.297217] Lustre: Skipped 1 previous similar message [ 9869.302488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9869.306586] Lustre: Skipped 7 previous similar messages [ 9875.463558] Lustre: server umount lustre-MDT0000 complete [ 9879.504886] LustreError: 331800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133420 with bad export cookie 5369659223429038200 [ 9879.505960] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9879.523718] LustreError: 331800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9880.022391] Lustre: server umount lustre-MDT0001 complete [ 9893.499813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9897.675912] Key type lgssc unregistered [ 9898.209036] LNet: 337201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9898.214543] LNetError: 337201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9898.231707] LNet: Removed LNI 192.168.204.145@tcp [ 9899.329170] Key type .llcrypt unregistered [ 9899.331748] Key type ._llcrypt unregistered [ 9976.822205] Key type ._llcrypt registered [ 9976.825345] Key type .llcrypt registered [ 9976.954897] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9992.386457] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9993.910159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9993.986365] alg: No test for adler32 (adler32-zlib) [ 9995.079948] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 9995.413436] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9997.111305] Key type lgssc registered [ 9998.101665] Lustre: Echo OBD driver; http://www.lustre.org/ [10031.389966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10031.400805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.576317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10032.606402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10032.660725] Lustre: lustre-MDT0000: new disk, initializing [10032.738327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10032.764201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10036.363902] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10050.299264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.470567] Lustre: 342139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10050.543077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10050.551250] Lustre: Skipped 1 previous similar message [10050.645209] Lustre: lustre-MDT0001: new disk, initializing [10050.790272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10050.869537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10050.890122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10055.824848] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10066.900275] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10075.009410] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10083.660691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.922842] Lustre: lustre-OST0000: new disk, initializing [10083.928929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10083.940561] Lustre: 344109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10084.042658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10089.569798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10089.578580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10089.750419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10091.309233] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10101.024551] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10107.667208] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10108.062621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10113.581397] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10113.865948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10120.674144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10120.678773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.687350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10120.695372] Lustre: Skipped 1 previous similar message [10120.707944] Lustre: Skipped 1 previous similar message [10125.796552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10125.814815] Lustre: Skipped 1 previous similar message [10130.399186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10130.588561] Lustre: server umount lustre-OST0000 complete [10136.039534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10136.064557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10136.070498] Lustre: Skipped 1 previous similar message [10140.468922] Lustre: server umount lustre-MDT0000 complete [10143.563822] LustreError: 342133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133684 with bad export cookie 6785387758005809034 [10143.567844] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10143.575154] LustreError: 342133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10143.824402] Lustre: server umount lustre-MDT0001 complete [10151.470563] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:34:51 (1789133691) [10159.328632] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:34:59 (1789133699) [10167.526351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.949890] LustreError: 346864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.973063] LustreError: 346864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10168.128546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10172.925557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10173.410306] LustreError: 346865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.427856] LustreError: 346865:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10177.507532] LustreError: 346864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10181.055556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.396541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10186.403871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10193.331568] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10200.441515] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10209.141880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10209.656061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10217.853145] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10225.195282] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10232.138117] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10233.968194] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10235.628110] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10237.727483] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10239.652573] Lustre: Failing over lustre-MDT0000 [10239.891844] Lustre: server umount lustre-MDT0000 complete [10241.506688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.510253] LustreError: 346860:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.523466] Lustre: Skipped 3 previous similar messages [10241.538396] LustreError: 346860:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10246.626666] LustreError: 348107:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.636464] LustreError: 348107:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10256.864710] LustreError: 346861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.882776] LustreError: 346861:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10257.887915] Lustre: 340360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789133782/real 1789133782] req@0000000061ee3108 x1876042486719232/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789133798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10257.941369] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10258.425647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.442086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10267.491793] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10269.233287] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10271.814054] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10272.768586] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10272.828665] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10272.913722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10281.978882] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10284.178661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10288.230672] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10289.991858] Lustre: Failing over lustre-MDT0000 [10290.244782] Lustre: server umount lustre-MDT0000 complete [10293.219059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10293.224141] LustreError: Skipped 1 previous similar message [10293.225239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.226106] LustreError: 346864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.226114] LustreError: 346864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [10309.159350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.424033] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10309.890208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10309.969380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10310.195106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10315.279455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10315.299602] Lustre: Skipped 2 previous similar messages [10315.351198] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10315.437418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10316.204830] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10328.240876] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10330.805858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10340.837479] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10340.862692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10340.863107] Lustre: Skipped 3 previous similar messages [10343.490789] Lustre: server umount lustre-MDT0000 complete [10345.953618] LustreError: 346861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.998559] LustreError: 346861:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [10347.807693] LustreError: 346845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133888 with bad export cookie 6785387758005811750 [10347.813237] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10347.832886] LustreError: 346845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10348.367670] Lustre: server umount lustre-MDT0001 complete [10363.021272] Lustre: server umount lustre-OST0000 complete [10382.850835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.760977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.631142] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10398.463135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.901341] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10412.648962] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10418.467790] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10426.889042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10428.605472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10436.002279] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10445.818442] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10454.497921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10454.500419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.501673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10454.501678] Lustre: Skipped 2 previous similar messages [10454.515109] Lustre: Skipped 2 previous similar messages [10456.881510] Lustre: server umount lustre-OST0000 complete [10460.639936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10460.643864] LustreError: Skipped 1 previous similar message [10464.739845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10464.746035] Lustre: Skipped 3 previous similar messages [10466.533188] Lustre: server umount lustre-MDT0000 complete [10469.864991] LustreError: 354499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.874468] LustreError: 354499:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10469.967192] LustreError: 353942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134010 with bad export cookie 6785387758005812499 [10469.978592] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10469.991063] LustreError: 353942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10470.401888] Lustre: server umount lustre-MDT0001 complete [10483.232418] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10487.390639] Key type lgssc unregistered [10487.900574] LNet: 356745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10487.919294] LNetError: 356745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10487.938252] LNet: Removed LNI 192.168.204.145@tcp [10489.079251] Key type .llcrypt unregistered [10489.081965] Key type ._llcrypt unregistered [10518.015827] Key type ._llcrypt registered [10518.027364] Key type .llcrypt registered [10518.241360] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10519.152314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10519.343197] alg: No test for adler32 (adler32-zlib) [10520.373486] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [10520.588760] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10522.247338] Key type lgssc registered [10523.410335] Lustre: Echo OBD driver; http://www.lustre.org/ [10531.897270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10531.925892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10533.449843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10538.790404] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10543.596616] LustreError: 358262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.621489] LustreError: 358262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10548.707552] LustreError: 358263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.080074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.743008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10555.113914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10564.811687] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10571.894543] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10581.207911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10582.025513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10591.344727] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10602.261261] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10603.518081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10618.849720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.857226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10618.865712] Lustre: Skipped 2 previous similar messages [10618.869582] Lustre: Skipped 1 previous similar message [10623.972287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10623.982666] Lustre: Skipped 3 previous similar messages [10625.298476] Lustre: server umount lustre-MDT0000 complete [10629.088588] LustreError: 358262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.103640] LustreError: 358242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134170 with bad export cookie 2487310517078735549 [10629.119121] LustreError: 358262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10629.126219] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10629.140978] LustreError: 358242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10629.593496] Lustre: server umount lustre-MDT0001 complete [10643.793255] Lustre: server umount lustre-OST0000 complete [10658.834341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.271901] LustreError: 361995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.351929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10664.417138] LustreError: 361996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.709726] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10674.098289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.683987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10680.331451] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10688.265969] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10693.723887] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10702.607406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10703.459952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10704.533334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10710.477952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10720.494099] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10735.600432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.606231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10735.625423] Lustre: Skipped 2 previous similar messages [10735.650231] Lustre: Skipped 2 previous similar messages [10738.932499] Lustre: server umount lustre-MDT0000 complete [10740.719782] LustreError: 361991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10740.744522] LustreError: 361991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10743.809193] LustreError: 362730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134284 with bad export cookie 2487310517078736662 [10743.812316] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10743.834180] LustreError: 362730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10744.405127] Lustre: server umount lustre-MDT0001 complete [10759.264048] Lustre: server umount lustre-OST0000 complete [10778.358435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.175847] LustreError: 365913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.331083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10784.005631] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10795.421776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.908793] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10809.687737] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10816.516349] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10825.496495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.071771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.088949] Lustre: Skipped 1 previous similar message [10833.972688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10843.772122] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10849.645765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10860.007030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.010548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10860.011208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10860.030027] Lustre: Skipped 1 previous similar message [10862.686518] Lustre: server umount lustre-OST0000 complete [10867.684725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10867.692854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.706610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10867.716175] Lustre: Skipped 1 previous similar message [10872.833838] Lustre: server umount lustre-MDT0000 complete [10875.363629] LustreError: 365914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.390983] LustreError: 365914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10876.621407] LustreError: 365895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134417 with bad export cookie 2487310517078737684 [10876.632049] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.636335] LustreError: 365895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10877.147270] Lustre: server umount lustre-MDT0001 complete [10889.261472] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10893.423402] Key type lgssc unregistered [10893.809869] LNet: 369463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10893.816947] LNetError: 369463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10894.886211] LNet: Removed LNI 192.168.204.145@tcp [10896.075897] Key type .llcrypt unregistered [10896.078446] Key type ._llcrypt unregistered [10924.504930] Key type ._llcrypt registered [10924.508816] Key type .llcrypt registered [10924.592350] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10925.992633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10926.019032] alg: No test for adler32 (adler32-zlib) [10927.036288] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [10927.196895] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10928.912648] Key type lgssc registered [10930.572316] Lustre: Echo OBD driver; http://www.lustre.org/ [10938.145825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10938.182214] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.195789] LustreError: 370906:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10938.222195] LustreError: 370906:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10938.240521] LustreError: 370906:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10938.249515] LustreError: 370906:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10938.256323] LustreError: 370906:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10949.243492] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.253401] LustreError: 371303:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10949.278558] LustreError: 371303:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10949.288713] LustreError: 371303:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10949.304273] LustreError: 371303:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10949.312312] LustreError: 371303:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10969.970077] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10972.760551] Key type lgssc unregistered [10973.004174] LNet: 372311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10973.010773] LNetError: 372311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10973.019666] LNet: Removed LNI 192.168.204.145@tcp [10973.908313] Key type .llcrypt unregistered [10973.915255] Key type ._llcrypt unregistered [10992.967574] Key type ._llcrypt registered [10992.970842] Key type .llcrypt registered [10993.112487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [11009.420688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11011.107894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11011.129792] alg: No test for adler32 (adler32-zlib) [11012.306679] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [11012.718226] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11014.423298] Key type lgssc registered [11015.612424] Lustre: Echo OBD driver; http://www.lustre.org/ [11054.067969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11054.078618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.320436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11055.373185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11055.520608] Lustre: lustre-MDT0000: new disk, initializing [11055.595504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11055.628290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11060.994552] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11072.570099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.718577] Lustre: 375471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11072.856420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11072.865169] Lustre: Skipped 1 previous similar message [11073.001745] Lustre: lustre-MDT0001: new disk, initializing [11073.154472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11073.234271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11073.247243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11078.285142] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11089.005220] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11096.128472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11103.966769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11104.498913] Lustre: lustre-OST0000: new disk, initializing [11104.525490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11104.535860] Lustre: 377441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11104.689418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11112.433803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11114.077853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11114.099894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11114.246265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11123.133804] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11129.162283] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11129.421414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11134.664947] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11134.915108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11140.068196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11140.082881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.091564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.102014] Lustre: Skipped 1 previous similar message [11140.138830] Lustre: Skipped 1 previous similar message [11143.770776] Lustre: server umount lustre-OST0000 complete [11149.799664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11149.807680] LustreError: Skipped 1 previous similar message [11149.812179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.825442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.874609] Lustre: server umount lustre-MDT0000 complete [11159.596453] LustreError: 375464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134700 with bad export cookie 17715623285685716060 [11159.625214] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11160.465842] Lustre: server umount lustre-MDT0001 complete [11169.565652] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11171.234520] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:51:51 (1789134711) [11178.321920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11178.989508] LustreError: 379901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.022792] LustreError: 379901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11179.123485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11184.618606] LustreError: 379902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.641731] LustreError: 379902:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11184.696300] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11189.730313] LustreError: 379901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.352412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.699412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11199.525363] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11208.562848] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11214.817433] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11223.967204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11224.345674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11232.839799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11241.812994] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11250.111659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.356465] Lustre: lustre-OST0001: new disk, initializing [11250.360172] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11250.367832] Lustre: 382738:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11250.469868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11256.475697] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11258.017619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11258.031135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11258.175170] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11267.342817] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11273.712555] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11273.722366] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11273.726554] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11273.755109] Lustre: Skipped 2 previous similar messages [11273.767420] Lustre: Skipped 3 previous similar messages [11277.823451] Lustre: server umount lustre-OST0001 complete [11278.832539] LustreError: 381724:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.873887] LustreError: 381724:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11283.943927] LustreError: 381725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.971757] LustreError: 381725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11294.178526] LustreError: 381724:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.204236] LustreError: 381724:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11297.280085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11297.301246] LustreError: Skipped 1 previous similar message [11297.313105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.341905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11297.349693] Lustre: Skipped 1 previous similar message [11304.425831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11304.436746] Lustre: Skipped 3 previous similar messages [11311.073530] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11311.377839] Lustre: server umount lustre-OST0000 complete [11317.728514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11317.742354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.780956] Lustre: Skipped 1 previous similar message [11317.784316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.797691] Lustre: Skipped 2 previous similar messages [11321.711805] Lustre: server umount lustre-MDT0000 complete [11324.896405] LustreError: 379896:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.909454] LustreError: 379896:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11325.525342] LustreError: 383775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134866 with bad export cookie 17715623285685716942 [11325.531516] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11325.540865] LustreError: 383775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11332.032447] Lustre: server umount lustre-MDT0001 complete [11352.654468] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11357.402619] Key type lgssc unregistered [11357.901400] LNet: 384758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11357.916519] LNetError: 384758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11357.939443] LNet: Removed LNI 192.168.204.145@tcp [11359.538346] Key type .llcrypt unregistered [11359.543891] Key type ._llcrypt unregistered [11426.247121] Key type ._llcrypt registered [11426.250428] Key type .llcrypt registered [11426.373485] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11428.093416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11428.136038] alg: No test for adler32 (adler32-zlib) [11429.407379] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [11429.752976] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11431.391159] Key type lgssc registered [11433.217896] Lustre: Echo OBD driver; http://www.lustre.org/ [11441.490413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11441.506829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11442.839311] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11442.890407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11443.104583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11448.152491] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11463.420272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11463.704328] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11463.782021] Lustre: 387562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11464.392327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11471.620496] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11479.041937] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11487.107837] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11491.813472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11491.820095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11491.828822] Lustre: Skipped 1 previous similar message [11496.883952] Lustre: server umount lustre-MDT0000 complete [11502.050040] LustreError: 387573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.081149] LustreError: 387573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11502.113843] LustreError: 387554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135043 with bad export cookie 12157119856387714436 [11502.127026] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11502.135242] LustreError: 387554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11502.911978] Lustre: server umount lustre-MDT0001 complete [11514.648348] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:57:34 (1789135054) [11522.437437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11523.134444] LustreError: 390077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.284758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11528.569812] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11528.681255] LustreError: 390078:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.798735] LustreError: 390077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11538.914963] LustreError: 390078:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.339393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.841350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11545.419780] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11555.597235] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11561.760513] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11569.772743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11570.002985] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11570.254791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11577.738858] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11586.573243] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11595.982668] Lustre: Permanently deactivating lustre-OST0000 [11596.013967] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11602.455606] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11603.657738] LustreError: 393017:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11603.667222] LustreError: 393017:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11603.673635] LustreError: 393017:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11606.096888] LustreError: 393065:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11606.105531] LustreError: 393065:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11614.337442] Lustre: server umount lustre-OST0000 complete [11616.912042] LustreError: 393314:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11616.916315] LustreError: 393314:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11621.856114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11621.861072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.869581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11621.877788] Lustre: Skipped 1 previous similar message [11622.883877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.908564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11622.917034] Lustre: Skipped 1 previous similar message [11628.002410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11628.021240] Lustre: Skipped 1 previous similar message [11629.176445] Lustre: server umount lustre-MDT0000 complete [11631.017954] LustreError: 391510:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.041529] LustreError: 391510:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11633.769612] LustreError: 390057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135174 with bad export cookie 12157119856387715017 [11633.777855] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11633.786198] LustreError: 390057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11643.555652] Lustre: server umount lustre-MDT0001 complete [11652.932646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.566801] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11663.958120] LustreError: 394477:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11671.322541] Lustre: 394816:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.145@tcp [11673.901819] Lustre: server umount MGS complete [11685.764475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.262181] LustreError: 395540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.278741] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11686.284192] Lustre: Skipped 1 previous similar message [11686.404782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11692.849068] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11703.610507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.120495] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11704.267780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11710.687977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11719.386923] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11726.379520] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11728.833463] Lustre: Permanently reactivating lustre-OST0000 [11728.858818] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11728.867546] Lustre: Skipped 2 previous similar messages [11758.562850] LustreError: 397380:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11768.899992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11769.509175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11776.795075] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11785.437942] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11793.183311] Lustre: server umount lustre-OST0000 complete [11799.530341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.552040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11799.567649] Lustre: Skipped 1 previous similar message [11799.589661] Lustre: Skipped 1 previous similar message [11807.804526] Lustre: server umount lustre-MDT0000 complete [11809.770153] LustreError: 397430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.777382] LustreError: 397430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11810.783326] Lustre: 386884:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135335/real 1789135335] req@00000000051ddba0 x1876043990405632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789135351 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11810.831708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.341197] LustreError: 395521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135352 with bad export cookie 12157119856387716067 [11811.348408] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11811.355852] LustreError: 395521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11811.670618] Lustre: server umount lustre-MDT0001 complete [11823.858485] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11826.914531] Key type lgssc unregistered [11827.235872] LNet: 399971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11827.248564] LNetError: 399971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11827.273767] LNet: Removed LNI 192.168.204.145@tcp [11828.447854] Key type .llcrypt unregistered [11828.450297] Key type ._llcrypt unregistered [11846.421704] Key type ._llcrypt registered [11846.423760] Key type .llcrypt registered [11846.500937] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [11860.717447] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11862.246074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11862.286978] alg: No test for adler32 (adler32-zlib) [11863.404980] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [11863.558616] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11865.215169] Key type lgssc registered [11866.340834] Lustre: Echo OBD driver; http://www.lustre.org/ [11916.550298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11916.579440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.921548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11917.989424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11918.064796] Lustre: lustre-MDT0000: new disk, initializing [11918.143332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11918.161037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11924.821815] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11937.139453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.257989] Lustre: 403129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11937.318756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11937.325863] Lustre: Skipped 1 previous similar message [11937.402384] Lustre: lustre-MDT0001: new disk, initializing [11937.476541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11937.502061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11937.519753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11942.513559] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11953.382613] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11960.515320] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.853056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11969.116930] Lustre: lustre-OST0000: new disk, initializing [11969.120380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11969.126240] Lustre: 405101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11969.198307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11974.771356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11974.780316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11974.861401] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11975.003668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11987.034275] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11992.898262] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11993.114817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12000.434911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12000.820890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12005.859031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12005.863661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.869593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12005.886815] LustreError: Skipped 1 previous similar message [12005.921576] Lustre: Skipped 1 previous similar message [12010.589693] Lustre: server umount lustre-OST0000 complete [12015.688557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12015.691461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12015.701965] Lustre: Skipped 1 previous similar message [12015.715073] Lustre: Skipped 2 previous similar messages [12020.705897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12020.718180] Lustre: Skipped 1 previous similar message [12022.298589] Lustre: server umount lustre-MDT0000 complete [12028.131132] LustreError: 405100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135569 with bad export cookie 15085435385553335309 [12028.132483] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.179233] LustreError: 405100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12028.802898] Lustre: server umount lustre-MDT0001 complete [12043.270578] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12045.593815] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:06:25 (1789135585) [12053.410897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12053.961507] LustreError: 407562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12053.994176] LustreError: 407562:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12054.094483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12059.107447] LustreError: 407563:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.129960] LustreError: 407563:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12059.874778] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12064.226123] LustreError: 407562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12069.347699] LustreError: 407563:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.075900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.503121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12075.606217] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12084.764261] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12091.804703] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12100.418263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.754589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12107.281379] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12114.418700] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12116.951704] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12129.255310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12141.027791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12141.028802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.045765] LustreError: Skipped 1 previous similar message [12141.049836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.081169] Lustre: Skipped 1 previous similar message [12141.125537] Lustre: Skipped 1 previous similar message [12142.546495] Lustre: server umount lustre-OST0000 complete [12151.271121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.297655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12153.798460] Lustre: server umount lustre-MDT0000 complete [12156.388713] LustreError: 407559:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.419065] LustreError: 407559:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12157.879819] LustreError: 407544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135698 with bad export cookie 15085435385553336191 [12157.893431] LustreError: 407544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12157.894297] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.142954] Lustre: server umount lustre-MDT0001 complete [12170.254386] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12173.576223] Key type lgssc unregistered [12173.938716] LNet: 411315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12173.951500] LNetError: 411315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12175.014547] LNet: Removed LNI 192.168.204.145@tcp [12176.183630] Key type .llcrypt unregistered [12176.188891] Key type ._llcrypt unregistered [12202.072699] Key type ._llcrypt registered [12202.074825] Key type .llcrypt registered [12202.190089] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12202.511703] Key type .llcrypt unregistered [12202.517901] Key type ._llcrypt unregistered [12220.825413] Key type ._llcrypt registered [12220.827534] Key type .llcrypt registered [12220.989870] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12222.653146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12222.665955] alg: No test for adler32 (adler32-zlib) [12223.783384] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12224.048645] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12225.787482] Key type lgssc registered [12227.096156] Lustre: Echo OBD driver; http://www.lustre.org/ [12236.467744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12236.508677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.318072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12244.477556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12248.547227] LustreError: 413542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.582911] LustreError: 413542:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12253.667309] LustreError: 413541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.643778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.088555] LustreError: 413542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.148113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.345176] LustreError: 413541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.359387] LustreError: 413541:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12262.229129] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12263.426677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12268.518312] LustreError: 413542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.532070] LustreError: 413542:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12269.675707] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12276.978146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.544290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12282.558219] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12293.089981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12293.107741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12293.148975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12293.170829] Lustre: Skipped 1 previous similar message [12298.222468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12298.233040] Lustre: Skipped 1 previous similar message [12303.343702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12303.353217] Lustre: Skipped 1 previous similar message [12305.376304] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12305.755158] Lustre: server umount lustre-OST0000 complete [12313.571609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.577420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12313.600057] Lustre: Skipped 1 previous similar message [12313.601074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.231778] Lustre: server umount lustre-MDT0000 complete [12318.691309] LustreError: 413541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.738795] LustreError: 413541:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12322.487016] LustreError: 413521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135863 with bad export cookie 10208224264045597493 [12322.498611] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12322.512269] LustreError: 413521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12323.224405] Lustre: server umount lustre-MDT0001 complete [12335.330572] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12338.360941] Key type lgssc unregistered [12338.678770] LNet: 416697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12338.692081] LNetError: 416697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12338.710673] LNet: Removed LNI 192.168.204.145@tcp [12339.624926] Key type .llcrypt unregistered [12339.626212] Key type ._llcrypt unregistered [12366.657959] Key type ._llcrypt registered [12366.666797] Key type .llcrypt registered [12366.798253] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12367.856432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12367.971546] alg: No test for adler32 (adler32-zlib) [12369.083688] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12369.264363] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12370.935347] Key type lgssc registered [12371.914166] Lustre: Echo OBD driver; http://www.lustre.org/ [12380.368519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12380.383270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12395.872038] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12400.992798] LustreError: 418089:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [12427.743622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12435.227620] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12444.439238] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12452.442331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12453.180566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12453.261540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12458.196147] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12458.465804] LustreError: 418115:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.483320] LustreError: 418115:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12462.566920] LustreError: 419127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12462.583973] LustreError: 419127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12467.612561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.694420] LustreError: 418115:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12467.709652] LustreError: 418115:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12468.081734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12472.599958] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12480.305593] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12486.259489] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12493.795382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12493.801690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.804026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12493.819463] Lustre: Skipped 1 previous similar message [12496.077720] Lustre: server umount lustre-OST0000 complete [12500.322691] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12500.333451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.354766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.364702] Lustre: Skipped 1 previous similar message [12500.779218] Lustre: server umount lustre-MDT0000 complete [12504.036138] LustreError: 420836:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12504.048500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.861275] LustreError: 418096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136045 with bad export cookie 9700954745713118868 [12504.867366] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12504.868779] LustreError: 418096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12507.656626] Lustre: server umount lustre-MDT0001 complete [12520.762554] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12523.943944] Key type lgssc unregistered [12524.264536] LNet: 421665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12524.270534] LNetError: 421665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12524.287543] LNet: Removed LNI 192.168.204.145@tcp [12525.491952] Key type .llcrypt unregistered [12525.499375] Key type ._llcrypt unregistered [12550.985466] Key type ._llcrypt registered [12550.988040] Key type .llcrypt registered [12551.116553] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12552.167382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12552.238583] alg: No test for adler32 (adler32-zlib) [12553.263344] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12553.476299] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12555.159200] Key type lgssc registered [12556.526230] Lustre: Echo OBD driver; http://www.lustre.org/ [12565.051575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12565.100760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.119620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12573.069498] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12577.269374] LustreError: 423088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12577.321299] LustreError: 423088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12581.297976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.921413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12587.484390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12594.630852] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12600.064070] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12606.214358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12606.380037] LustreError: 424909:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.502549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12612.600621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12613.336974] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12620.341433] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12627.941287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12627.946484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12628.032876] Lustre: Skipped 2 previous similar messages [12628.051198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12632.421444] Lustre: server umount lustre-MDT0000 complete [12633.059860] LustreError: 423083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12633.069334] LustreError: 423083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12634.762950] LustreError: 423069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1789136175 with bad export cookie 9338853503196569717 [12634.783591] LustreError: 423069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12637.212927] LustreError: 425665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136178 with bad export cookie 9338853503196568821 [12637.226904] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12637.229447] LustreError: 425665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12638.180651] LustreError: 423817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.184989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.215682] LustreError: 423817:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12638.237461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12638.271721] Lustre: Skipped 3 previous similar messages [12642.222554] LustreError: 423088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.232710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12642.252670] LustreError: 423088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12643.629207] Lustre: server umount lustre-OST0000 complete [12651.034046] Lustre: server umount lustre-MDT0001 complete [12663.356327] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12666.574288] Key type lgssc unregistered [12667.022633] LNet: 426776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12667.037702] LNetError: 426776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12667.066475] LNet: Removed LNI 192.168.204.145@tcp [12668.117216] Key type .llcrypt unregistered [12668.120269] Key type ._llcrypt unregistered [12696.004108] Key type ._llcrypt registered [12696.006442] Key type .llcrypt registered [12696.129768] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12697.365480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12697.389342] alg: No test for adler32 (adler32-zlib) [12698.637473] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12698.871869] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12700.543462] Key type lgssc registered [12702.279272] Lustre: Echo OBD driver; http://www.lustre.org/ [12711.128804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12711.147140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12712.500836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12717.123166] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12722.677117] LustreError: 428210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.692569] LustreError: 428210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12726.030241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12726.331456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12731.595428] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12739.552828] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12746.002723] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12753.533594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12753.975888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12757.052397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12760.489595] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12768.710298] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12775.475054] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [12776.421705] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12776.439953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12776.444806] Lustre: Skipped 1 previous similar message [12778.835668] Lustre: server umount lustre-MDT0001 complete [12780.601905] LustreError: 428205:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.660586] LustreError: 428210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.661197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12786.693709] LustreError: 428210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12786.716905] Lustre: Skipped 1 previous similar message [12786.735441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12791.440572] Lustre: server umount lustre-OST0000 complete [12791.778323] LustreError: 428209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.534704] Lustre: server umount lustre-MDT0000 complete [12811.506786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12814.569222] Key type lgssc unregistered [12814.906860] LNet: 431896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12814.919514] LNetError: 431896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12814.936442] LNet: Removed LNI 192.168.204.145@tcp [12815.718499] Key type .llcrypt unregistered [12815.721763] Key type ._llcrypt unregistered [12842.193969] Key type ._llcrypt registered [12842.195673] Key type .llcrypt registered [12842.363668] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [12858.007575] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12859.214644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12859.247818] alg: No test for adler32 (adler32-zlib) [12860.224314] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12860.546932] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12862.295558] Key type lgssc registered [12863.716574] Lustre: Echo OBD driver; http://www.lustre.org/ [12903.879845] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12920.831748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12920.854493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.087840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12922.161036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12922.298826] Lustre: lustre-MDT0000: new disk, initializing [12922.458964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12922.499396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12927.755818] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12943.643550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12943.963677] Lustre: lustre-OST0000: new disk, initializing [12943.968911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12943.973883] Lustre: Skipped 1 previous similar message [12943.980795] Lustre: 436931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12944.041256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12951.100687] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12952.171539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12952.182269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12952.247364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12960.931881] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12977.512802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12977.690432] Lustre: 435915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12977.784128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12977.899624] Lustre: lustre-MDT0001: new disk, initializing [12978.048632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12978.092611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12978.103602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12982.975845] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12993.898678] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12994.244867] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12999.996946] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13000.279442] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13005.292114] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.295825] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13005.315444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13010.402817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13010.414839] Lustre: Skipped 3 previous similar messages [13011.542502] Lustre: server umount lustre-MDT0001 complete [13020.641590] LustreError: 435925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.662099] LustreError: 435925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [13025.767755] LustreError: 435921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.807989] LustreError: 435921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13030.884322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.892679] LustreError: 435926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13030.903366] Lustre: Skipped 1 previous similar message [13030.906536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13030.926511] Lustre: Skipped 1 previous similar message [13036.001260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13036.009373] LustreError: 435925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.191139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13044.194467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13044.197337] LustreError: 435926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.197349] LustreError: 435926:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13044.236309] Lustre: Skipped 1 previous similar message [13044.413839] Lustre: server umount lustre-OST0000 complete [13054.689941] Lustre: server umount lustre-MDT0000 complete [13068.123621] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13070.723043] Key type lgssc unregistered [13071.099490] LNet: 440686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13071.110670] LNetError: 440686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13071.153558] LNet: Removed LNI 192.168.204.145@tcp [13072.017584] Key type .llcrypt unregistered [13072.018855] Key type ._llcrypt unregistered [13113.487632] Key type ._llcrypt registered [13113.490251] Key type .llcrypt registered [13113.601896] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13113.974857] Key type .llcrypt unregistered [13113.979150] Key type ._llcrypt unregistered [13129.581803] Key type ._llcrypt registered [13129.585257] Key type .llcrypt registered [13129.734207] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13130.922320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13130.939975] alg: No test for adler32 (adler32-zlib) [13132.148210] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [13132.509592] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [13134.231176] Key type lgssc registered [13135.479798] Lustre: Echo OBD driver; http://www.lustre.org/ [13161.271477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13161.326658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13162.563858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13162.600317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13162.674135] Lustre: lustre-MDT0000: new disk, initializing [13162.787184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13162.804020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13167.418676] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13178.576618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.709952] Lustre: 444068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13178.768562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13178.771920] Lustre: Skipped 1 previous similar message [13178.829430] Lustre: lustre-MDT0001: new disk, initializing [13178.929487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.971523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13178.987951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13183.504737] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13192.599966] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13197.653895] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13204.431499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13204.714666] Lustre: lustre-OST0000: new disk, initializing [13204.726945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13204.734395] Lustre: 446038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13204.831359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13210.885272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13214.273134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13214.285267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13214.398462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13219.645201] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13230.050943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13230.055755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13230.058771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13230.065975] Lustre: Skipped 2 previous similar messages [13232.737801] Lustre: server umount lustre-MDT0000 complete [13236.499024] LustreError: 444861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136777 with bad export cookie 4841044266212690761 [13236.507215] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13236.514039] LustreError: 444861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13236.945119] Lustre: server umount lustre-MDT0001 complete [13251.042202] Lustre: server umount lustre-OST0000 complete [13263.533660] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:26:43 (1789136803) [13271.821675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13272.266466] LustreError: 448066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13272.298964] LustreError: 448066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13272.386176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13277.674426] LustreError: 448067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13277.715487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13282.792949] LustreError: 448066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13286.227973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13286.551958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13290.947283] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13297.941905] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13303.801797] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13323.885080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13324.202605] Lustre: Found index 0 for lustre-OST0000, updating log [13324.454188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13332.170636] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13340.264433] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13343.927678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13353.444628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13353.450684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13353.477797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13353.484980] Lustre: Skipped 2 previous similar messages [13354.531979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13354.538413] Lustre: Skipped 1 previous similar message [13360.610955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13360.616848] Lustre: Skipped 1 previous similar message [13365.215529] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13365.418148] Lustre: server umount lustre-OST0000 complete [13382.123694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13382.127477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13382.134145] Lustre: Skipped 2 previous similar messages [13382.143548] Lustre: Skipped 1 previous similar message [13383.391435] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13387.118526] Lustre: server umount lustre-MDT0000 complete [13387.232272] LustreError: 448067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13387.257150] LustreError: 448067:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13391.333879] LustreError: 448066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13391.359558] LustreError: 448066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13391.577353] LustreError: 449909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136932 with bad export cookie 4841044266212692154 [13391.590899] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13391.607231] LustreError: 449909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13392.002754] Lustre: server umount lustre-MDT0001 complete [13402.332538] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:29:02 (1789136942) [13446.879621] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14025.228361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14025.806463] LustreError: 454480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14025.882652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14031.334466] LustreError: 454481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14031.439321] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14035.426501] LustreError: 454480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14040.929916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14041.202406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14046.759368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14055.492837] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14061.620176] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14082.366581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14082.541662] Lustre: Found index 0 for lustre-OST0000, updating log [14082.784759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14087.904941] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14089.147332] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14099.193486] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14102.252091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.266247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14107.555888] Lustre: server umount lustre-OST0000 complete [14112.614420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14112.641083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14112.736535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14114.789967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14114.809845] Lustre: Skipped 4 previous similar messages [14118.891138] LustreError: 454480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14118.917105] LustreError: 454480:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14119.023690] Lustre: server umount lustre-MDT0000 complete [14123.473137] LustreError: 456340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137664 with bad export cookie 4841044266212693239 [14123.500945] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14123.512432] LustreError: 456340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14124.132970] Lustre: server umount lustre-MDT0001 complete [14150.497527] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [14156.049391] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14166.744448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14204.440077] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:42:24 (1789137744) [14223.609256] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14227.122087] Key type lgssc unregistered [14227.444917] LNet: 461106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14227.455378] LNetError: 461106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14345.253807] LNet: Removed LNI 192.168.204.145@tcp [14346.246785] Key type .llcrypt unregistered [14346.252289] Key type ._llcrypt unregistered [14363.846743] Key type ._llcrypt registered [14363.848050] Key type .llcrypt registered [14363.914207] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14364.876437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14364.963412] alg: No test for adler32 (adler32-zlib) [14366.294625] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [14366.538554] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14368.194092] Key type lgssc registered [14369.328734] Lustre: Echo OBD driver; http://www.lustre.org/ [14374.292475] LNet: Added LNI 192.168.204.145@tcp100 [8/256/0/180] [14375.377533] LNet: Added LNI 192.168.204.145@tcp101 [8/256/0/180] [14376.341796] LNet: Added LNI 192.168.204.145@tcp102 [8/256/0/180] [14377.260954] LNet: Added LNI 192.168.204.145@tcp103 [8/256/0/180] [14378.314737] LNet: Added LNI 192.168.204.145@tcp104 [8/256/0/180] [14379.083678] LNet: Added LNI 192.168.204.145@tcp105 [8/256/0/180] [14380.089344] LNet: Added LNI 192.168.204.145@tcp106 [8/256/0/180] [14381.071994] LNet: Added LNI 192.168.204.145@tcp107 [8/256/0/180] [14382.300058] LNet: Added LNI 192.168.204.145@tcp108 [8/256/0/180] [14383.408342] LNet: Added LNI 192.168.204.145@tcp109 [8/256/0/180] [14384.471195] LNet: Added LNI 192.168.204.145@tcp110 [8/256/0/180] [14385.525779] LNet: Added LNI 192.168.204.145@tcp111 [8/256/0/180] [14386.648854] LNet: Added LNI 192.168.204.145@tcp112 [8/256/0/180] [14387.389530] LNet: Added LNI 192.168.204.145@tcp113 [8/256/0/180] [14388.189773] LNet: Added LNI 192.168.204.145@tcp114 [8/256/0/180] [14389.013257] LNet: Added LNI 192.168.204.145@tcp115 [8/256/0/180] [14389.942046] LNet: Added LNI 192.168.204.145@tcp116 [8/256/0/180] [14391.060756] LNet: Added LNI 192.168.204.145@tcp117 [8/256/0/180] [14392.245463] LNet: Added LNI 192.168.204.145@tcp118 [8/256/0/180] [14393.261392] LNet: Added LNI 192.168.204.145@tcp119 [8/256/0/180] [14394.106298] LNet: Added LNI 192.168.204.145@tcp120 [8/256/0/180] [14394.982556] LNet: Added LNI 192.168.204.145@tcp121 [8/256/0/180] [14396.009405] LNet: Added LNI 192.168.204.145@tcp122 [8/256/0/180] [14396.824876] LNet: Added LNI 192.168.204.145@tcp123 [8/256/0/180] [14397.522985] LNet: Added LNI 192.168.204.145@tcp124 [8/256/0/180] [14398.187315] LNet: Added LNI 192.168.204.145@tcp125 [8/256/0/180] [14398.997677] LNet: Added LNI 192.168.204.145@tcp126 [8/256/0/180] [14399.897222] LNet: Added LNI 192.168.204.145@tcp127 [8/256/0/180] [14400.760077] LNet: Added LNI 192.168.204.145@tcp128 [8/256/0/180] [14401.573220] LNet: Added LNI 192.168.204.145@tcp129 [8/256/0/180] [14402.376356] LNet: Added LNI 192.168.204.145@tcp130 [8/256/0/180] [14403.089088] LNet: Added LNI 192.168.204.145@tcp131 [8/256/0/180] [14403.984923] LNet: Added LNI 192.168.204.145@tcp132 [8/256/0/180] [14404.939568] LNet: Added LNI 192.168.204.145@tcp133 [8/256/0/180] [14405.847377] LNet: Added LNI 192.168.204.145@tcp134 [8/256/0/180] [14406.727457] LNet: Added LNI 192.168.204.145@tcp135 [8/256/0/180] [14413.564508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14413.628664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14414.729845] Lustre: 464141:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14414.806583] LNetError: 464168:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14414.820538] LNetError: Protocol error connecting to 192.168.204.145@tcp119 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14414.833153] LNetError: Skipped 5 previous similar messages [14414.922049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14414.950948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14415.039653] Lustre: lustre-MDT0000: new disk, initializing [14415.143160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14415.164358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14415.711426] LNet: 8 local NIs in recovery (showing 8): 192.168.204.145@tcp123, 192.168.204.145@tcp126, 192.168.204.145@tcp127, 192.168.204.145@tcp119, 192.168.204.145@tcp120, 192.168.204.145@tcp128, 192.168.204.145@tcp125, 192.168.204.145@tcp129 [14415.713065] LNetError: 464164:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14415.761714] LNetError: 464164:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14415.787510] LNetError: Protocol error connecting to 192.168.204.145@tcp100 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14415.798564] LNetError: Skipped 5 previous similar messages [14419.743352] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14430.624037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14430.775666] Lustre: 464159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14430.831912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14430.837211] Lustre: Skipped 1 previous similar message [14430.903656] Lustre: lustre-MDT0001: new disk, initializing [14430.934415] Lustre: 464959:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14430.942879] Lustre: 464959:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14431.056873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14431.092121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14431.100287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14435.602204] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14445.418784] LNetError: 461597:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14445.457467] LNetError: 461597:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14445.474865] LNetError: Protocol error connecting to 192.168.204.145@tcp100 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14446.512689] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14447.474967] LNetError: 464161:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14447.495849] LNetError: Protocol error connecting to 192.168.204.145@tcp124 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14451.571031] LNetError: 464168:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14451.582510] LNetError: Protocol error connecting to 192.168.204.145@tcp117 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14453.224884] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14460.833751] LNetError: 461600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14460.848690] LNetError: 461600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14460.867579] LNetError: Protocol error connecting to 192.168.204.145@tcp124 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14460.892148] LNetError: Skipped 2 previous similar messages [14462.056739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14462.445855] Lustre: lustre-OST0000: new disk, initializing [14462.449573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14462.455380] Lustre: 466134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14462.553222] Lustre: 466139:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14462.558869] Lustre: 466139:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14462.617241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14469.525845] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14471.735617] Lustre: 466591:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14471.751384] Lustre: 466591:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14471.820259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14471.835583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14471.957681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14477.218821] LNetError: 464164:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14477.228058] LNetError: 464164:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14477.234645] LNetError: Protocol error connecting to 192.168.204.145@tcp117 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14477.255363] LNetError: Skipped 4 previous similar messages [14479.302510] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14487.547236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14487.567678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14487.596106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14492.646165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14492.653337] Lustre: Skipped 1 previous similar message [14497.761724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14497.782827] Lustre: Skipped 1 previous similar message [14501.343137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14501.673760] Lustre: server umount lustre-OST0000 complete [14508.000716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14508.004628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14508.006471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14508.006479] Lustre: Skipped 1 previous similar message [14508.019194] Lustre: Skipped 2 previous similar messages [14508.048371] LustreError: Skipped 1 previous similar message [14511.009589] LNetError: 461600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14511.020447] LNetError: 461600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14511.028813] LNetError: Protocol error connecting to 192.168.204.145@tcp124 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14511.037144] LNetError: Skipped 5 previous similar messages [14512.442979] Lustre: server umount lustre-MDT0000 complete [14516.479734] LustreError: 464149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138057 with bad export cookie 1127569430862631759 [14516.487786] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14516.810449] Lustre: server umount lustre-MDT0001 complete [14527.994070] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14530.875700] Key type lgssc unregistered [14531.282758] LNet: 468052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14531.287814] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14531.299339] LNet: Removed LNI 192.168.204.145@tcp [14532.319267] LNet: Removed LNI 192.168.204.145@tcp100 [14532.324495] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14532.336611] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14532.347932] LNet: Removed LNI 192.168.204.145@tcp101 [14533.407162] LNet: Removed LNI 192.168.204.145@tcp102 [14533.420839] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14533.437970] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14533.447171] LNet: Removed LNI 192.168.204.145@tcp103 [14534.495300] LNet: Removed LNI 192.168.204.145@tcp104 [14535.519180] LNet: Removed LNI 192.168.204.145@tcp105 [14535.524662] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14535.531476] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14535.544068] LNet: Removed LNI 192.168.204.145@tcp106 [14536.608347] LNet: Removed LNI 192.168.204.145@tcp107 [14537.631228] LNet: Removed LNI 192.168.204.145@tcp108 [14538.655879] LNet: Removed LNI 192.168.204.145@tcp109 [14539.679184] LNet: Removed LNI 192.168.204.145@tcp110 [14539.681580] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14539.690130] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14539.701578] LNet: Removed LNI 192.168.204.145@tcp111 [14540.767758] LNet: Removed LNI 192.168.204.145@tcp112 [14541.791197] LNet: Removed LNI 192.168.204.145@tcp113 [14542.815437] LNet: Removed LNI 192.168.204.145@tcp114 [14543.839266] LNet: Removed LNI 192.168.204.145@tcp115 [14544.863855] LNet: Removed LNI 192.168.204.145@tcp116 [14545.887863] LNet: Removed LNI 192.168.204.145@tcp117 [14546.911171] LNet: Removed LNI 192.168.204.145@tcp118 [14547.935289] LNet: Removed LNI 192.168.204.145@tcp119 [14547.938079] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14547.950932] LNetError: 468052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14547.964534] LNet: Removed LNI 192.168.204.145@tcp120 [14549.023423] LNet: Removed LNI 192.168.204.145@tcp121 [14550.047198] LNet: Removed LNI 192.168.204.145@tcp122 [14551.076082] LNet: Removed LNI 192.168.204.145@tcp123 [14552.095446] LNet: Removed LNI 192.168.204.145@tcp124 [14553.119414] LNet: Removed LNI 192.168.204.145@tcp125 [14554.143180] LNet: Removed LNI 192.168.204.145@tcp126 [14555.167201] LNet: Removed LNI 192.168.204.145@tcp127 [14556.191930] LNet: Removed LNI 192.168.204.145@tcp128 [14557.215250] LNet: Removed LNI 192.168.204.145@tcp129 [14557.224710] LNet: Removed LNI 192.168.204.145@tcp130 [14557.235836] LNet: Removed LNI 192.168.204.145@tcp131 [14557.246205] LNet: Removed LNI 192.168.204.145@tcp132 [14557.252732] LNet: Removed LNI 192.168.204.145@tcp133 [14557.262979] LNet: Removed LNI 192.168.204.145@tcp134 [14557.278673] LNet: Removed LNI 192.168.204.145@tcp135 [14558.252550] Key type .llcrypt unregistered [14558.255310] Key type ._llcrypt unregistered [14585.798982] Key type ._llcrypt registered [14585.804614] Key type .llcrypt registered [14585.973711] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14603.315361] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14605.010869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14605.057982] alg: No test for adler32 (adler32-zlib) [14606.239883] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [14606.742990] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14608.504718] Key type lgssc registered [14609.797324] Lustre: Echo OBD driver; http://www.lustre.org/ [14650.792383] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:49:50 (1789138190) [14671.301939] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14674.874232] Key type lgssc unregistered [14675.244205] LNet: 472150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14675.252987] LNetError: 472150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14675.268156] LNet: Removed LNI 192.168.204.145@tcp [14676.223378] Key type .llcrypt unregistered [14676.227365] Key type ._llcrypt unregistered [14693.925839] Key type ._llcrypt registered [14693.927254] Key type .llcrypt registered [14694.049153] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14694.978934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14695.054261] alg: No test for adler32 (adler32-zlib) [14696.292326] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [14696.533569] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14698.183570] Key type lgssc registered [14699.347350] Lustre: Echo OBD driver; http://www.lustre.org/ [14710.089664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14710.121260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14711.596339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14711.682296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14711.812985] Lustre: lustre-MDT0000: new disk, initializing [14711.967370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14712.021103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14717.349368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14729.181853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14729.447596] Lustre: 473363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14729.515850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14729.523365] Lustre: Skipped 1 previous similar message [14729.664951] Lustre: lustre-MDT0001: new disk, initializing [14729.805975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14729.906974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14729.931927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14735.260827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14745.801665] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14752.376160] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14759.451298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14759.850517] Lustre: lustre-OST0000: new disk, initializing [14759.855120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14759.863166] Lustre: 475334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14759.984711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14766.143827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14766.151389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14766.253286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14766.430725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14777.248539] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14782.223407] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lnet_if_list [14789.295476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14789.328847] LNet: Added LNI 192.168.204.145@tcp10 [8/256/0/180] [14795.711181] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14795.737424] LNet: Added LNI 192.168.204.145@tcp11 [8/256/0/180] [14800.882221] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14800.916662] LNet: Added LNI 192.168.204.145@tcp12 [8/256/0/180] [14806.592191] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14806.630726] LNet: Added LNI 192.168.204.145@tcp13 [8/256/0/180] [14812.032991] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14812.060515] LNet: Added LNI 192.168.204.145@tcp14 [8/256/0/180] [14817.326955] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14817.351396] LNet: Added LNI 192.168.204.145@tcp15 [8/256/0/180] [14823.274486] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14823.312626] LNet: Added LNI 192.168.204.145@tcp16 [8/256/0/180] [14830.924847] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14830.975828] LNet: Added LNI 192.168.204.145@tcp17 [8/256/0/180] [14837.750490] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14837.787938] LNet: Added LNI 192.168.204.145@tcp18 [8/256/0/180] [14844.406861] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14844.472813] LNet: Added LNI 192.168.204.145@tcp19 [8/256/0/180] [14851.525040] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14851.578287] LNet: Added LNI 192.168.204.145@tcp20 [8/256/0/180] [14858.565686] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14858.606631] LNet: Added LNI 192.168.204.145@tcp21 [8/256/0/180] [14864.974864] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14865.003974] LNet: Added LNI 192.168.204.145@tcp22 [8/256/0/180] [14872.015205] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14872.054218] LNet: Added LNI 192.168.204.145@tcp23 [8/256/0/180] [14879.353873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14879.404715] LNet: Added LNI 192.168.204.145@tcp24 [8/256/0/180] [14887.483432] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14887.543083] LNet: Added LNI 192.168.204.145@tcp25 [8/256/0/180] [14895.525689] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14895.559039] LNet: Added LNI 192.168.204.145@tcp26 [8/256/0/180] [14902.859438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14902.901643] LNet: Added LNI 192.168.204.145@tcp27 [8/256/0/180] [14911.776728] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14911.822329] LNet: Added LNI 192.168.204.145@tcp28 [8/256/0/180] [14919.971895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14920.026490] LNet: Added LNI 192.168.204.145@tcp29 [8/256/0/180] [14928.572481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14928.602903] LNet: Added LNI 192.168.204.145@tcp30 [8/256/0/180] [14937.251867] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14937.309197] LNet: Added LNI 192.168.204.145@tcp31 [8/256/0/180] [14946.906401] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14946.949485] LNet: Added LNI 192.168.204.145@tcp32 [8/256/0/180] [14955.807631] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14955.869880] LNet: Added LNI 192.168.204.145@tcp33 [8/256/0/180] [14964.941765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14964.995381] LNet: Added LNI 192.168.204.145@tcp34 [8/256/0/180] [14972.434888] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14972.464754] LNet: Added LNI 192.168.204.145@tcp35 [8/256/0/180] [14979.958556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14979.992439] LNet: Added LNI 192.168.204.145@tcp36 [8/256/0/180] [14987.545611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14987.575850] LNet: Added LNI 192.168.204.145@tcp37 [8/256/0/180] [14995.621601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14995.673572] LNet: Added LNI 192.168.204.145@tcp38 [8/256/0/180] [15003.692703] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15003.731955] LNet: Added LNI 192.168.204.145@tcp39 [8/256/0/180] [15011.562838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15011.628049] LNet: Added LNI 192.168.204.145@tcp40 [8/256/0/180] [15020.361259] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15020.400874] LNet: Added LNI 192.168.204.145@tcp41 [8/256/0/180] [15029.419950] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15029.459498] LNet: Added LNI 192.168.204.145@tcp42 [8/256/0/180] [15038.974942] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15039.010483] LNet: Added LNI 192.168.204.145@tcp43 [8/256/0/180] [15048.070322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15048.098977] LNet: Added LNI 192.168.204.145@tcp44 [8/256/0/180] [15056.281751] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15056.311579] LNet: Added LNI 192.168.204.145@tcp45 [8/256/0/180] [15066.055909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15066.149527] LNet: Added LNI 192.168.204.145@tcp46 [8/256/0/180] [15075.628852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15075.670203] LNet: Added LNI 192.168.204.145@tcp47 [8/256/0/180] [15084.183885] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15084.218755] LNet: Added LNI 192.168.204.145@tcp48 [8/256/0/180] [15093.319583] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15093.377470] LNet: Added LNI 192.168.204.145@tcp49 [8/256/0/180] [15101.925833] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15101.955194] LNet: Added LNI 192.168.204.145@tcp50 [8/256/0/180] [15123.937308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15123.946745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15123.974283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15128.041191] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15128.042191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15128.071320] Lustre: Skipped 2 previous similar messages [15129.764243] Lustre: server umount lustre-MDT0000 complete [15133.503639] LustreError: 473356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138674 with bad export cookie 16133942844734921400 [15133.508958] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15133.520418] LustreError: 473356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15133.848453] Lustre: server umount lustre-MDT0001 complete [15153.148330] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15154.655316] Lustre: 472635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138679/real 1789138679] req@00000000968f179c x1876047416200960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15154.699170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15154.724982] Lustre: Skipped 1 previous similar message [15159.775719] Lustre: 472637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138684/real 1789138684] req@00000000c3226608 x1876047416201216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15160.944725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15164.959182] Lustre: 472637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138689/real 1789138689] req@0000000082989547 x1876047416201344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15170.015826] Lustre: 472634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138694/real 1789138694] req@000000006db2b173 x1876047416201472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15178.273719] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdfe7478936e1657e [15178.286283] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [15178.610979] LustreError: 475326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15178.625716] LustreError: 475326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [15178.663628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15179.680518] LustreError: 499584:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15179.692196] LustreError: 499584:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15183.175266] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15183.788223] LustreError: 499582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15183.802509] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15183.815911] LustreError: 499582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15188.972110] LustreError: 499583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15188.991226] LustreError: 499583:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15191.458022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15191.838758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15196.815212] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15197.162160] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15205.197786] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15211.213638] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15232.993507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15233.004394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15233.007934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15233.025351] Lustre: Skipped 1 previous similar message [15233.034636] Lustre: Skipped 1 previous similar message [15238.117622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15238.139714] Lustre: Skipped 1 previous similar message [15243.235253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15243.249202] Lustre: Skipped 1 previous similar message [15245.279193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15245.514969] Lustre: server umount lustre-OST0000 complete [15253.472780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15253.478665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15253.508610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15253.542959] Lustre: Skipped 1 previous similar message [15256.128737] Lustre: server umount lustre-MDT0000 complete [15258.594757] LustreError: 499587:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15260.590301] LustreError: 473358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138801 with bad export cookie 16133942844734924158 [15260.595956] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15260.609253] LustreError: 473358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15261.185950] Lustre: server umount lustre-MDT0001 complete [15274.147984] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15277.270490] Key type lgssc unregistered [15277.739504] LNet: 502389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15277.747308] LNetError: 502389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15277.752437] LNet: Removed LNI 192.168.204.145@tcp [15277.755568] LNet: Removed LNI 192.168.204.145@tcp10 [15277.758521] LNet: Removed LNI 192.168.204.145@tcp11 [15277.764492] LNet: Removed LNI 192.168.204.145@tcp12 [15277.767674] LNet: Removed LNI 192.168.204.145@tcp13 [15277.771296] LNet: Removed LNI 192.168.204.145@tcp14 [15277.773393] LNet: Removed LNI 192.168.204.145@tcp15 [15277.778428] LNet: Removed LNI 192.168.204.145@tcp16 [15277.782397] LNet: Removed LNI 192.168.204.145@tcp17 [15277.785408] LNet: Removed LNI 192.168.204.145@tcp18 [15277.787851] LNet: Removed LNI 192.168.204.145@tcp19 [15277.790974] LNet: Removed LNI 192.168.204.145@tcp20 [15277.794356] LNet: Removed LNI 192.168.204.145@tcp21 [15277.796392] LNet: Removed LNI 192.168.204.145@tcp22 [15277.798248] LNet: Removed LNI 192.168.204.145@tcp23 [15277.802281] LNet: Removed LNI 192.168.204.145@tcp24 [15277.806635] LNet: Removed LNI 192.168.204.145@tcp25 [15277.812784] LNet: Removed LNI 192.168.204.145@tcp26 [15277.815682] LNet: Removed LNI 192.168.204.145@tcp27 [15277.823022] LNet: Removed LNI 192.168.204.145@tcp28 [15277.826952] LNet: Removed LNI 192.168.204.145@tcp29 [15277.832406] LNet: Removed LNI 192.168.204.145@tcp30 [15277.840608] LNet: Removed LNI 192.168.204.145@tcp31 [15277.849610] LNet: Removed LNI 192.168.204.145@tcp32 [15277.851432] LNet: Removed LNI 192.168.204.145@tcp33 [15277.861965] LNet: Removed LNI 192.168.204.145@tcp34 [15277.869377] LNet: Removed LNI 192.168.204.145@tcp35 [15277.872423] LNet: Removed LNI 192.168.204.145@tcp36 [15277.879882] LNet: Removed LNI 192.168.204.145@tcp37 [15277.888427] LNet: Removed LNI 192.168.204.145@tcp38 [15277.895451] LNet: Removed LNI 192.168.204.145@tcp39 [15277.898415] LNet: Removed LNI 192.168.204.145@tcp40 [15277.903478] LNet: Removed LNI 192.168.204.145@tcp41 [15277.909450] LNet: Removed LNI 192.168.204.145@tcp42 [15277.915650] LNet: Removed LNI 192.168.204.145@tcp43 [15277.924251] LNet: Removed LNI 192.168.204.145@tcp44 [15277.928035] LNet: Removed LNI 192.168.204.145@tcp45 [15277.931896] LNet: Removed LNI 192.168.204.145@tcp46 [15277.939304] LNet: Removed LNI 192.168.204.145@tcp47 [15277.942650] LNet: Removed LNI 192.168.204.145@tcp48 [15277.947236] LNet: Removed LNI 192.168.204.145@tcp49 [15277.966655] LNet: Removed LNI 192.168.204.145@tcp50 [15278.872908] Key type .llcrypt unregistered [15278.876079] Key type ._llcrypt unregistered [15301.145657] Key type ._llcrypt registered [15301.148761] Key type .llcrypt registered [15301.260611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15317.299167] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15318.430324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15318.553853] alg: No test for adler32 (adler32-zlib) [15319.664404] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [15319.961759] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15321.775230] Key type lgssc registered [15322.978367] Lustre: Echo OBD driver; http://www.lustre.org/ [15361.667410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15361.716680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15363.041808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15363.116538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15363.195827] Lustre: lustre-MDT0000: new disk, initializing [15363.297906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15363.338482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15368.115278] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15379.711732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15379.935985] Lustre: 505597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15380.036229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15380.041649] Lustre: Skipped 1 previous similar message [15380.147794] Lustre: lustre-MDT0001: new disk, initializing [15380.245707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15380.289393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15380.308347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15386.166718] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15397.169828] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15404.019240] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15412.473404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15412.837317] Lustre: *** cfs_fail_loc=90f, val=0*** [15412.941313] Lustre: lustre-OST0000: new disk, initializing [15412.945216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15412.952340] Lustre: 507614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15413.039623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15421.047171] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15421.999514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15422.016367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15422.284446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15431.938592] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15442.912802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15442.916076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15442.925620] LustreError: Skipped 1 previous similar message [15442.927807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15442.942522] Lustre: Skipped 1 previous similar message [15448.036340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15448.047198] Lustre: Skipped 2 previous similar messages [15453.153982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15453.158928] Lustre: Skipped 1 previous similar message [15454.180940] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15454.311521] Lustre: server umount lustre-OST0000 complete [15458.277633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15458.281593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15458.283105] Lustre: Skipped 1 previous similar message [15458.305806] Lustre: Skipped 1 previous similar message [15463.395056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15463.406675] Lustre: Skipped 1 previous similar message [15463.699163] Lustre: server umount lustre-MDT0000 complete [15467.148987] LustreError: 505589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139008 with bad export cookie 17167840118341619680 [15467.152918] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15467.164924] LustreError: 505589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15467.584359] Lustre: server umount lustre-MDT0001 complete [15480.291633] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15483.609313] Key type lgssc unregistered [15483.917692] LNet: 509526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15483.931728] LNetError: 509526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15483.943210] LNet: Removed LNI 192.168.204.145@tcp [15485.064640] Key type .llcrypt unregistered [15485.066609] Key type ._llcrypt unregistered [15504.153994] Key type ._llcrypt registered [15504.157634] Key type .llcrypt registered [15504.287454] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15505.683860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15505.727600] alg: No test for adler32 (adler32-zlib) [15506.791742] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [15507.106113] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15508.791177] Key type lgssc registered [15509.906656] Lustre: Echo OBD driver; http://www.lustre.org/ [15518.719608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15518.742751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15520.218311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15524.542617] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15530.472221] LustreError: 510799:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15530.484060] LustreError: 510799:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15532.519095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15532.992802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15537.720451] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15545.104817] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15550.357312] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15557.192705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15557.555738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15563.427175] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15571.458265] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15580.947127] loop0: detected capacity change from 0 to 204800000 [15581.225881] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15581.235975] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15581.245747] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15587.447794] loop0: detected capacity change from 0 to 204800000 [15587.663200] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15587.691736] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15587.719052] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15593.034266] loop0: detected capacity change from 0 to 204800000 [15597.716517] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15597.815437] Lustre: fs15246-OST0000: new disk, initializing [15597.818178] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15597.822623] Lustre: 514389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15597.869069] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15603.637270] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15614.371254] loop1: detected capacity change from 0 to 204800000 [15620.123902] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15620.228525] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15620.279631] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15620.344655] Lustre: fs15246-MDT0000: new disk, initializing [15620.481293] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15620.528808] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15620.663818] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15621.728621] Lustre: 515512:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000028b1e3e x1876048266442112/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789139178 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15625.846605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15630.819902] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15630.833307] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15630.885470] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15648.225944] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15648.249165] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15653.351842] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15654.292799] Lustre: server umount fs15246-MDT0000 complete [15658.474298] LustreError: 510796:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15661.988367] Lustre: server umount fs15246-OST0000 complete [15671.265754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15671.275068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15671.282460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15673.312832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15673.322137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15673.333236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15676.123628] Lustre: server umount lustre-OST0000 complete [15681.508708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15681.514845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15681.546388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15681.559109] Lustre: Skipped 1 previous similar message [15694.304748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15694.313672] Lustre: Skipped 4 previous similar messages [15695.328666] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15695.579105] Lustre: server umount lustre-MDT0000 complete [15698.403245] LustreError: 513093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15698.417629] LustreError: 513093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15700.101353] LustreError: 510781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139241 with bad export cookie 3049809079615744043 [15700.103743] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15700.119657] LustreError: 510781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15700.519455] Lustre: server umount lustre-MDT0001 complete [15713.162847] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15716.663795] Key type lgssc unregistered [15717.022602] LNet: 518613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15717.028418] LNetError: 518613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15717.047051] LNet: Removed LNI 192.168.204.145@tcp [15718.110534] Key type .llcrypt unregistered [15718.114280] Key type ._llcrypt unregistered [15737.851254] Key type ._llcrypt registered [15737.853594] Key type .llcrypt registered [15737.971662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15738.875492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15739.059500] alg: No test for adler32 (adler32-zlib) [15740.071334] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [15740.322924] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15742.039169] Key type lgssc registered [15743.023738] Lustre: Echo OBD driver; http://www.lustre.org/ [15782.926850] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15794.543622] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15829.177403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15829.199672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15830.451880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15830.481532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15830.565947] Lustre: lustre-MDT0000: new disk, initializing [15830.642627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15830.664721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15834.733972] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15844.517320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15844.634974] Lustre: 523085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15844.694338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15844.698361] Lustre: Skipped 1 previous similar message [15844.768762] Lustre: lustre-MDT0001: new disk, initializing [15844.858633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15844.889769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15844.898934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15849.842993] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15859.529308] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15865.190898] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15872.036280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15872.422157] Lustre: lustre-OST0000: new disk, initializing [15872.426816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15872.434544] Lustre: 525051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15872.522898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15879.782162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15879.792155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15880.004890] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15880.019121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15889.724830] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15895.614949] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15895.891413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15901.198084] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15901.410840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15905.765128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15905.769164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15905.770577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15905.790553] Lustre: Skipped 1 previous similar message [15905.832590] LustreError: Skipped 1 previous similar message [15910.173268] Lustre: server umount lustre-OST0000 complete [15916.002466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15916.003959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15916.022951] Lustre: Skipped 1 previous similar message [15916.035079] Lustre: Skipped 1 previous similar message [15921.121578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15921.132411] Lustre: Skipped 1 previous similar message [15925.219108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15925.227802] Lustre: Skipped 1 previous similar message [15928.287139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15928.564306] Lustre: server umount lustre-MDT0000 complete [15932.544670] LustreError: 523887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139473 with bad export cookie 1406543064734280326 [15932.547752] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15932.554666] LustreError: 523887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15932.938521] Lustre: server umount lustre-MDT0001 complete [15956.660403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15966.620772] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15996.108775] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15695 sec ========= 11:12:15 (1789139535) [15997.869088] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:12:17 (1789139537) === [16002.051463] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:12:21 (1789139541) === [16041.473013] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [16044.208935] Key type lgssc unregistered [16044.488769] LNet: 531700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16044.494281] LNetError: 531700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16044.506061] LNet: Removed LNI 192.168.204.145@tcp [16045.249685] Key type .llcrypt unregistered [16045.254346] Key type ._llcrypt unregistered