[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486689717 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003009] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007022] pid_max: default: 32768 minimum: 301 [ 0.008157] LSM: Security Framework initializing [ 0.009061] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.020592] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024735] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027141] *** VALIDATE tmpfs *** [ 0.028479] *** VALIDATE proc *** [ 0.030029] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.031000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033037] Spectre V2 : User space: Vulnerable [ 0.034010] Speculative Store Bypass: Vulnerable [ 0.037925] debug: unmapping init [mem 0xffffffff86859000-0xffffffff86860fff] [ 0.040254] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042031] ... version: 2 [ 0.043018] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046020] ... max period: 00007fffffffffff [ 0.047020] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.050263] rcu: Hierarchical SRCU implementation. [ 0.052757] smp: Bringing up secondary CPUs ... [ 0.053725] x86: Booting SMP configuration: [ 0.054030] .... node #0, CPUs: #1 #2 #3 [ 0.064485] smp: Brought up 1 node, 4 CPUs [ 0.066019] smpboot: Max logical packages: 1 [ 0.067030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102000] node 0 deferred pages initialised in 33ms [ 0.107313] devtmpfs: initialized [ 0.108426] x86/mm: Memory block size: 128MB [ 0.110952] gcov: version magic: 0x41383552 [ 0.112966] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114378] pinctrl core: initialized pinctrl subsystem [ 0.115529] [ 0.116010] ************************************************************* [ 0.117016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118021] ** ** [ 0.119022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120016] ** ** [ 0.121020] ** This means that this kernel is built to expose internal ** [ 0.122019] ** IOMMU data structures, which may compromise security on ** [ 0.123015] ** your system. ** [ 0.124020] ** ** [ 0.125022] ** If you see this message and you are not debugging the ** [ 0.126150] ** kernel, report this immediately to your vendor! ** [ 0.127014] ** ** [ 0.128017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129016] ************************************************************* [ 0.130937] NET: Registered protocol family 16 [ 0.131475] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.132085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.133078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134829] cpuidle: using governor menu [ 0.136092] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137556] PCI: Using configuration type 1 for base access [ 0.138235] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.146133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.147021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.149289] cryptd: max_cpu_qlen set to 1000 [ 0.154525] ACPI: Added _OSI(Module Device) [ 0.156019] ACPI: Added _OSI(Processor Device) [ 0.158011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.160012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.167062] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.175114] ACPI: Interpreter enabled [ 0.176059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.177020] ACPI: Using IOAPIC for interrupt routing [ 0.178095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.179444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.191321] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.192046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.193015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.194092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.196575] acpiphp: Slot [2] registered [ 0.197077] acpiphp: Slot [5] registered [ 0.198124] acpiphp: Slot [6] registered [ 0.199148] acpiphp: Slot [7] registered [ 0.200108] acpiphp: Slot [8] registered [ 0.201089] acpiphp: Slot [9] registered [ 0.202077] acpiphp: Slot [10] registered [ 0.203074] acpiphp: Slot [3] registered [ 0.204148] acpiphp: Slot [4] registered [ 0.205110] acpiphp: Slot [11] registered [ 0.206098] acpiphp: Slot [12] registered [ 0.207094] acpiphp: Slot [13] registered [ 0.208105] acpiphp: Slot [14] registered [ 0.209066] acpiphp: Slot [15] registered [ 0.210070] acpiphp: Slot [16] registered [ 0.211149] acpiphp: Slot [17] registered [ 0.212097] acpiphp: Slot [18] registered [ 0.213159] acpiphp: Slot [19] registered [ 0.214206] acpiphp: Slot [20] registered [ 0.215055] acpiphp: Slot [21] registered [ 0.216090] acpiphp: Slot [22] registered [ 0.217087] acpiphp: Slot [23] registered [ 0.218096] acpiphp: Slot [24] registered [ 0.219109] acpiphp: Slot [25] registered [ 0.220260] acpiphp: Slot [26] registered [ 0.221097] acpiphp: Slot [27] registered [ 0.222107] acpiphp: Slot [28] registered [ 0.223136] acpiphp: Slot [29] registered [ 0.223956] acpiphp: Slot [30] registered [ 0.224058] acpiphp: Slot [31] registered [ 0.224880] PCI host bridge to bus 0000:00 [ 0.225023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.226016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.227029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.228020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.229023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.230023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.231195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.233301] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.235252] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.241016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.244058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.245026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.246024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.247027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.248579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.249516] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.250029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.251653] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.256016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.269015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.273012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.280203] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.292024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.312030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.369020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.379021] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455033] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.470020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.484014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.507015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518823] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.536019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.585050] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.594017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.602031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682092] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.692017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.698014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.712035] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.780463] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.791854] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.796082] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.799571] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.804286] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.811006] iommu: Default domain type: Passthrough [ 0.825707] SCSI subsystem initialized [ 0.827174] ACPI: bus type USB registered [ 0.829116] usbcore: registered new interface driver usbfs [ 0.831205] usbcore: registered new interface driver hub [ 0.833081] usbcore: registered new device driver usb [ 0.835182] pps_core: LinuxPPS API ver. 1 registered [ 0.837013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.841197] PTP clock support registered [ 0.845113] EDAC MC: Ver: 3.0.0 [ 0.849259] PCI: Using ACPI for IRQ routing [ 0.853200] NetLabel: Initializing [ 0.855010] NetLabel: domain hash size = 128 [ 0.857012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.859101] NetLabel: unlabeled traffic allowed by default [ 0.862169] vgaarb: loaded [ 0.864374] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.868036] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.874242] clocksource: Switched to clocksource kvm-clock [ 0.998825] VFS: Disk quotas dquot_6.6.0 [ 1.000080] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.002111] *** VALIDATE ramfs *** [ 1.003119] *** VALIDATE hugetlbfs *** [ 1.004269] pnp: PnP ACPI init [ 1.007647] pnp: PnP ACPI: found 6 devices [ 1.027273] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.029929] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.031967] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.033953] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.036092] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.038080] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.040421] NET: Registered protocol family 2 [ 1.042789] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.047580] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.051834] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.057526] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.060633] TCP: Hash tables configured (established 65536 bind 65536) [ 1.063460] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.066865] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070080] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072817] NET: Registered protocol family 1 [ 1.075687] RPC: Registered named UNIX socket transport module. [ 1.077468] RPC: Registered udp transport module. [ 1.078730] RPC: Registered tcp transport module. [ 1.080017] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081536] NET: Registered protocol family 44 [ 1.083098] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.084338] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.086080] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.088370] PCI: CLS 0 bytes, default 64 [ 1.091202] Unpacking initramfs... [ 2.967366] debug: unmapping init [mem 0xffff8bd3fcc54000-0xffff8bd3fffbffff] [ 2.971599] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.973732] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.976866] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.599528] Initialise system trusted keyrings [ 3.601545] Key type blacklist registered [ 3.603673] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.627987] zbud: loaded [ 3.709309] *** VALIDATE nfs *** [ 3.711072] *** VALIDATE nfs4 *** [ 3.713827] pstore: using deflate compression [ 3.719903] Platform Keyring initialized [ 3.834565] NET: Registered protocol family 38 [ 3.836113] Key type asymmetric registered [ 3.837362] Asymmetric key parser 'x509' registered [ 3.839151] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.842031] io scheduler mq-deadline registered [ 3.843414] io scheduler kyber registered [ 3.844917] io scheduler bfq registered [ 3.846984] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.849685] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.852363] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.854908] ACPI: Power Button [PWRF] [ 3.861094] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.869645] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.882319] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.889665] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.905302] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.933265] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.967492] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.971974] Non-volatile memory driver v1.3 [ 3.973572] Linux agpgart interface v0.103 [ 3.998128] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 4.000930] vda: detected capacity change from 0 to 71311360 [ 4.014081] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.016771] vdb: detected capacity change from 0 to 1073741824 [ 4.036293] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.039294] vdc: detected capacity change from 0 to 2621440000 [ 4.057195] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.059859] vdd: detected capacity change from 0 to 2621440000 [ 4.077352] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.080308] vde: detected capacity change from 0 to 4294967296 [ 4.095720] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.098634] vdf: detected capacity change from 0 to 4294967296 [ 4.107385] libphy: Fixed MDIO Bus: probed [ 4.113168] usbcore: registered new interface driver usbserial_generic [ 4.116697] usbserial: USB Serial support registered for generic [ 4.118783] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.123134] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.124969] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.127222] mousedev: PS/2 mouse device common for all mice [ 4.129763] rtc_cmos 00:05: RTC can wake from S4 [ 4.132681] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.133918] rtc_cmos 00:05: registered as rtc0 [ 4.138314] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.140890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.144628] intel_pstate: CPU model not supported [ 4.150932] hid: raw HID events driver (C) Jiri Kosina [ 4.152302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.155058] usbcore: registered new interface driver usbhid [ 4.155064] usbhid: USB HID core driver [ 4.155194] drop_monitor: Initializing network drop monitor service [ 4.155333] Initializing XFRM netlink socket [ 4.159690] NET: Registered protocol family 10 [ 4.169799] Segment Routing with IPv6 [ 4.171522] NET: Registered protocol family 17 [ 4.173696] mpls_gso: MPLS GSO support [ 4.178622] RAS: Correctable Errors collector initialized. [ 4.180864] AVX version of gcm_enc/dec engaged. [ 4.182524] AES CTR mode by8 optimization enabled [ 4.297263] sched_clock: Marking stable (4297205196, 0)->(5359351635, -1062146439) [ 4.302196] registered taskstats version 1 [ 4.304983] Loading compiled-in X.509 certificates [ 4.307857] zswap: loaded using pool lzo/zbud [ 4.353678] Key type big_key registered [ 4.374569] Key type encrypted registered [ 4.376167] ima: No TPM chip found, activating TPM-bypass! [ 4.378391] ima: Allocated hash algorithm: sha1 [ 4.380322] ima: No architecture policies found [ 4.382380] evm: Initialising EVM extended attributes: [ 4.384364] evm: security.selinux [ 4.385611] evm: security.ima [ 4.386759] evm: security.capability [ 4.388276] evm: HMAC attrs: 0x1 [ 4.391465] rtc_cmos 00:05: setting system clock to 2026-06-16 14:57:43 UTC (1781621863) [ 4.399360] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 4.402952] debug: unmapping init [mem 0xffffffff86582000-0xffffffff86858fff] [ 4.412158] Write protecting the kernel read-only data: 28672k [ 4.416721] debug: unmapping init [mem 0xffffffff84c03000-0xffffffff84dfffff] [ 4.419833] debug: unmapping init [mem 0xffffffff85514000-0xffffffff855fffff] [ 4.492323] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.501952] systemd[1]: Detected virtualization kvm. [ 4.503867] systemd[1]: Detected architecture x86-64. [ 4.506289] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.538638] systemd[1]: No hostname configured. [ 4.540834] systemd[1]: Set hostname to . [ 4.543732] random: systemd: uninitialized urandom read (16 bytes read) [ 4.546592] systemd[1]: Initializing machine ID from random generator. [ 4.701893] random: ln: uninitialized urandom read (6 bytes read) [ 4.920658] random: systemd: uninitialized urandom read (16 bytes read) [ 4.923948] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.936073] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.958620] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.135531] device-mapper: uevent: version 1.0.3 [ 7.137812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] [ 8.055394] random: fast init done Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.543850] virtio_net virtio0 ens2: renamed from eth0 [ 9.002725] scsi host0: ata_piix [ 9.404196] scsi host1: ata_piix [ 9.405563] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.407759] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.141532] random: crng init done [ 14.149519] random: 7 urandom warning(s) missed due to ratelimiting [ 16.862707] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.054673] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.835306] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.236195] SELinux: Disabled at runtime. [ 21.302714] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.310897] systemd[1]: Detected virtualization kvm. [ 21.312304] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.204509] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.209404] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.222855] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.229376] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.237158] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.251846] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.302144] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [[ 22.360102] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.551505] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.177039] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.668085] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.738336] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.936598] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 33.053133] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit)[ 33.560277] NFS: Registering the id_resolver key type [ 33.562819] Key type id_resolver registered [ 33.564857] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg256-server login: [ 63.541499] hrtimer: interrupt took 6059725 ns [ 107.851797] libcfs: loading out-of-tree module taints kernel. [ 107.896185] Key type ._llcrypt registered [ 107.899297] Key type .llcrypt registered [ 107.986370] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 121.262135] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 122.352478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.367778] alg: No test for adler32 (adler32-zlib) [ 123.647203] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 124.249329] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 125.943210] Key type lgssc registered [ 127.118705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.826471] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.745238] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 180.822539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.847238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.016592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.045383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.099222] Lustre: lustre-MDT0000: new disk, initializing [ 182.163191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.190271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.944231] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.371624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.444418] Lustre: 6506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.468226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.472844] Lustre: Skipped 1 previous similar message [ 195.531193] Lustre: lustre-MDT0001: new disk, initializing [ 195.583624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.614634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.624290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.285762] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.431945] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.462658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.690202] Lustre: lustre-OST0000: new disk, initializing [ 210.694886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.709325] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.777462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.682392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.694408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.800342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 215.688469] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.762604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.896790] Lustre: lustre-OST0001: new disk, initializing [ 227.901298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.906602] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.001319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.209899] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.562802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.573861] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.621048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.400795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.457362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.311813] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.523929] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing yml_node [ 258.244740] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 260.241108] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 262.127639] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 263.322857] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 11:02:01 EDT 2026 [ 275.428396] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 276.718493] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 291.809227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.815345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.820663] Lustre: Skipped 2 previous similar messages [ 291.835750] Lustre: Skipped 3 previous similar messages [ 296.930445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.933958] Lustre: Skipped 3 previous similar messages [ 297.132432] Lustre: server umount lustre-MDT0000 complete [ 299.933072] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781622159 with bad export cookie 6533783570300918168 [ 299.935753] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.946513] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 302.048841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.058356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.061535] Lustre: Skipped 2 previous similar messages [ 302.074206] Lustre: Skipped 1 previous similar message [ 304.096033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.102850] Lustre: Skipped 1 previous similar message [ 309.217959] LustreError: 11416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.218074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.245237] LustreError: 11416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 309.264634] Lustre: Skipped 1 previous similar message [ 310.240201] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.257032] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 314.335129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 314.696795] Lustre: server umount lustre-MDT0001 complete [ 318.329771] Lustre: server umount lustre-OST0000 complete [ 321.296514] Lustre: server umount lustre-OST0001 complete [ 327.007236] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 340.942757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 396.672330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.226870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.286349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.411697] Lustre: lustre-MDT0000: new disk, initializing [ 397.526514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.545607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.460317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 418.200117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.345348] Lustre: 15079:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 418.492489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 418.502155] Lustre: Skipped 1 previous similar message [ 418.634486] Lustre: lustre-MDT0001: new disk, initializing [ 418.766870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.818504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.835785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 423.975039] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 435.089372] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 441.854487] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 450.252519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.747676] Lustre: lustre-OST0000: new disk, initializing [ 450.760236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.779033] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.914906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.200128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.217115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.398801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.213519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 467.602806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 473.454950] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.656792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.809429] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.965303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.296617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.298957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.313545] LustreError: Skipped 1 previous similar message [ 483.322192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.348997] Lustre: Skipped 3 previous similar messages [ 495.071155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 495.286067] Lustre: server umount lustre-OST0000 complete [ 500.712188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.720271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.736552] Lustre: Skipped 1 previous similar message [ 500.746436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.750611] Lustre: Skipped 4 previous similar messages [ 505.632749] Lustre: server umount lustre-MDT0000 complete [ 508.904485] LustreError: 15088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.583131] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781622368 with bad export cookie 6533783570300919701 [ 509.591124] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.604500] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 509.902813] Lustre: server umount lustre-MDT0001 complete [ 514.432205] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:06:12 (1781622372) [ 517.120988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 518.918246] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:06:16 (1781622376) [ 525.998834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.565338] LustreError: 19461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.572624] LustreError: 19461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 526.625823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.960912] LustreError: 19462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.120294] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 541.154600] LustreError: 19461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.171822] LustreError: 19461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 542.184742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.249986] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 556.851729] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 562.748925] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.905897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.512114] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 585.972322] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 603.113738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.121159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.133173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.133182] Lustre: Skipped 2 previous similar messages [ 603.174348] Lustre: Skipped 2 previous similar messages [ 604.383623] Lustre: server umount lustre-OST0000 complete [ 609.257272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 609.273707] LustreError: Skipped 1 previous similar message [ 613.345860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.369756] Lustre: Skipped 1 previous similar message [ 615.317489] Lustre: server umount lustre-MDT0000 complete [ 618.464376] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.478215] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 620.154438] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781622479 with bad export cookie 6533783570300920583 [ 620.163643] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.165315] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 620.769282] Lustre: server umount lustre-MDT0001 complete [ 633.166638] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 636.221305] Key type lgssc unregistered [ 636.550042] LNet: 23008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.564779] LNetError: 23008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.606564] LNet: Removed LNI 192.168.202.156@tcp [ 638.812409] Key type .llcrypt unregistered [ 638.818379] Key type ._llcrypt unregistered [ 663.197655] Key type ._llcrypt registered [ 663.199955] Key type .llcrypt registered [ 663.308355] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 664.508946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.671065] alg: No test for adler32 (adler32-zlib) [ 665.642934] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 665.803547] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 667.519169] Key type lgssc registered [ 668.679763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.520208] loop: module loaded [ 677.622478] loop0: detected capacity change from 0 to 204800000 [ 677.897250] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.922933] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.218920] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.527550] loop0: detected capacity change from 0 to 204800000 [ 684.740174] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.758982] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.768232] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.181982] loop0: detected capacity change from 0 to 204800000 [ 696.553988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.578736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.866382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 697.931534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 698.006491] Lustre: lustre-MDT0000: new disk, initializing [ 698.167550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.198108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 702.307125] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 712.862415] loop1: detected capacity change from 0 to 204800000 [ 719.480960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.880876] Lustre: lustre-OST07c6: new disk, initializing [ 719.888385] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 719.894345] Lustre: Skipped 1 previous similar message [ 719.901772] Lustre: 26219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 720.013565] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 727.656507] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 728.758776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 728.770371] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 728.816407] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 733.979874] Lustre: 26227:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 733.988671] Lustre: 26227:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 733.992572] Lustre: 26227:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 733.999190] Lustre: 26227:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 734.009264] Lustre: 26227:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 734.024579] Lustre: 26227:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 749.868963] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 749.884125] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.888829] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.897814] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 749.909643] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 749.916212] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.173162] Lustre: 25204:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 755.194109] Lustre: 25204:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.206774] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.217495] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.228884] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.235330] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.243412] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 755.247931] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.255087] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.262164] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.268741] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.278328] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.549354] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 758.558026] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.564306] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 758.572781] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.578325] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.584967] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.589824] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 758.596046] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.600761] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.604624] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.608285] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.615190] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.674635] Lustre: 25204:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 763.694037] Lustre: 25204:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.714720] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.732807] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.745736] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.754508] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.764681] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 763.772521] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.783568] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.790829] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.796391] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.810346] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.287394] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 772.297496] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 772.304733] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.310931] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.312847] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.316017] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.320490] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 772.324101] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.326983] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.329614] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.332540] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.335791] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.945495] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.970275] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 836.065899] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.186447] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 841.190854] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.330183] Lustre: server umount lustre-OST07c6 complete [ 851.601560] Lustre: server umount lustre-MDT0000 complete [ 869.643177] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 872.650866] Key type lgssc unregistered [ 872.987217] LNet: 28788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.012871] LNetError: 28788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.032266] LNet: Removed LNI 192.168.202.156@tcp [ 873.863383] Key type .llcrypt unregistered [ 873.866485] Key type ._llcrypt unregistered [ 890.685207] Key type ._llcrypt registered [ 890.686957] Key type .llcrypt registered [ 890.773078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 891.644186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.656891] alg: No test for adler32 (adler32-zlib) [ 892.853706] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 893.068484] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 894.719163] Key type lgssc registered [ 895.945752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.900951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.930108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.450905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.520899] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 914.920963] LustreError: 30052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.940940] LustreError: 30052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 919.205270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.614661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.143766] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 931.903750] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 938.414075] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 945.905552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 946.455150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.596614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 953.666278] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 962.231947] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 981.475772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.489954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.530955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.339807] Lustre: server umount lustre-OST0000 complete [ 990.692570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.708098] Lustre: Skipped 1 previous similar message [ 990.712745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.727756] Lustre: Skipped 1 previous similar message [ 994.085656] Lustre: server umount lustre-MDT0000 complete [ 995.810796] LustreError: 30053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.838382] LustreError: 30053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 998.215241] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781622857 with bad export cookie 14947345703882051663 [ 998.224665] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.226887] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1008.744798] Lustre: server umount lustre-MDT0001 complete [ 1022.042548] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1025.159994] Key type lgssc unregistered [ 1025.456924] LNet: 33790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.465949] LNetError: 33790:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.535624] LNet: Removed LNI 192.168.202.156@tcp [ 1027.630246] Key type .llcrypt unregistered [ 1027.632244] Key type ._llcrypt unregistered [ 1047.321106] Key type ._llcrypt registered [ 1047.324187] Key type .llcrypt registered [ 1047.417842] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1048.913339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.961369] alg: No test for adler32 (adler32-zlib) [ 1049.995339] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1050.175453] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1051.919610] Key type lgssc registered [ 1053.199230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.592996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.605871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.380955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.441655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1073.640094] LustreError: 35060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.659877] LustreError: 35060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1077.198442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.597630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.745496] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1089.492213] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1095.472990] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.715403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.937963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.089720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1108.986887] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1116.159972] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1133.024625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.031636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.043929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1133.073665] Lustre: Skipped 1 previous similar message [ 1138.147463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.151702] Lustre: Skipped 2 previous similar messages [ 1139.063418] Lustre: server umount lustre-OST0000 complete [ 1143.264741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.277108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.285143] Lustre: Skipped 1 previous similar message [ 1143.292878] Lustre: Skipped 1 previous similar message [ 1148.385513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.396803] Lustre: Skipped 1 previous similar message [ 1148.785992] Lustre: server umount lustre-MDT0000 complete [ 1152.307953] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623011 with bad export cookie 8297401763677072170 [ 1152.318684] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.326160] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1152.721676] Lustre: server umount lustre-MDT0001 complete [ 1162.382320] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1164.892129] Key type lgssc unregistered [ 1165.267208] LNet: 38613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.271215] LNetError: 38613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.280575] LNet: Removed LNI 192.168.202.156@tcp [ 1165.998214] Key type .llcrypt unregistered [ 1166.002676] Key type ._llcrypt unregistered [ 1190.657199] Key type ._llcrypt registered [ 1190.658964] Key type .llcrypt registered [ 1190.736819] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1191.661396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.750046] alg: No test for adler32 (adler32-zlib) [ 1192.758199] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1192.961850] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1194.655177] Key type lgssc registered [ 1195.722672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1203.629374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1203.654224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.033068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.669857] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1215.459021] LustreError: 40031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.475087] LustreError: 40031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1217.526927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.895637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1222.808378] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1230.373538] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1236.273461] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1243.276984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.740644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.662073] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1249.958020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1257.965681] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1267.683029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.684224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.696700] Lustre: Skipped 1 previous similar message [ 1267.709467] Lustre: Skipped 2 previous similar messages [ 1270.955957] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1272.806310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.816184] Lustre: Skipped 2 previous similar messages [ 1273.845888] Lustre: server umount lustre-MDT0000 complete [ 1276.075397] LustreError: 42157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.887122] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623136 with bad export cookie 3197901473441226865 [ 1277.906470] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.926770] LustreError: 40028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.931708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.954015] Lustre: Skipped 1 previous similar message [ 1277.963823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1281.172057] LustreError: 40028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.044380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1283.051057] Lustre: Skipped 1 previous similar message [ 1286.285863] LustreError: 42157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.302476] LustreError: 42157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1291.414683] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 1291.421969] Lustre: Skipped 2 previous similar messages [ 1292.255129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1292.558565] Lustre: server umount lustre-MDT0001 complete [ 1307.244941] Lustre: server umount lustre-OST0000 complete [ 1323.152598] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1325.767399] Key type lgssc unregistered [ 1326.022429] LNet: 43893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.028949] LNetError: 43893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.085205] LNet: Removed LNI 192.168.202.156@tcp [ 1327.935631] Key type .llcrypt unregistered [ 1327.937921] Key type ._llcrypt unregistered [ 1348.103949] Key type ._llcrypt registered [ 1348.105933] Key type .llcrypt registered [ 1348.214811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1348.964281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.992090] alg: No test for adler32 (adler32-zlib) [ 1349.983671] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1350.179860] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1351.793377] Key type lgssc registered [ 1352.836487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.078077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.098909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.625525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.078689] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1371.112281] LustreError: 45289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.124184] LustreError: 45289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1374.006798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.355163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.077359] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1385.994430] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1391.233806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1397.269616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.459726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.898659] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1406.736768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1411.634250] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.934734] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1420.265487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.280711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.296274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.308946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.312779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1422.325553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.707286] Lustre: server umount lustre-OST0000 complete [ 1440.736302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1440.746101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.762582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.765370] Lustre: Skipped 1 previous similar message [ 1442.788961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.005660] Lustre: server umount lustre-MDT0000 complete [ 1447.909028] LustreError: 45286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.920852] LustreError: 45286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1449.971458] LustreError: 45271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623309 with bad export cookie 14984860721215241220 [ 1449.976711] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.985216] LustreError: 45271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1464.287133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1464.518891] Lustre: server umount lustre-MDT0001 complete [ 1473.455723] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1475.445448] Key type lgssc unregistered [ 1475.691728] LNet: 48998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.698640] LNetError: 48998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.720769] LNet: Removed LNI 192.168.202.156@tcp [ 1476.330231] Key type .llcrypt unregistered [ 1476.332685] Key type ._llcrypt unregistered [ 1495.148453] Key type ._llcrypt registered [ 1495.150331] Key type .llcrypt registered [ 1495.237093] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1496.139059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.183902] alg: No test for adler32 (adler32-zlib) [ 1497.192985] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1497.498858] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1499.168604] Key type lgssc registered [ 1500.034533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1506.865461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1506.878309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.228717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.966787] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1518.564256] LustreError: 50393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.577358] LustreError: 50393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1520.036678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.290912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1524.252646] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1531.092714] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1536.034397] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1542.071982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1542.500421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.824553] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1553.784495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1554.083969] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1559.927729] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1562.501946] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1574.431383] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1611h23m51s [ 1575.891852] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1579.490794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.493470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.504380] Lustre: Skipped 1 previous similar message [ 1579.516777] Lustre: Skipped 2 previous similar messages [ 1580.579376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.582341] Lustre: Skipped 2 previous similar messages [ 1582.261769] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1583.935437] Lustre: server umount lustre-MDT0000 complete [ 1586.659082] LustreError: 50394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.677600] LustreError: 50394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1587.122385] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1588.377460] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.804352] LustreError: 50393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.812646] LustreError: 50393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1592.910188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.996915] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.172945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.232043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1594.214939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1596.466642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1599.320908] Lustre: DEBUG MARKER: Wait for df (36568) ... [ 1600.601945] Lustre: DEBUG MARKER: done [ 1604.583412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.589784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1604.604689] Lustre: Skipped 2 previous similar messages [ 1604.615024] Lustre: Skipped 1 previous similar message [ 1609.699956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.711425] Lustre: Skipped 1 previous similar message [ 1609.851446] Lustre: server umount lustre-OST0000 complete [ 1614.816574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1614.824102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.846322] Lustre: Skipped 1 previous similar message [ 1620.326602] Lustre: server umount lustre-MDT0000 complete [ 1620.963710] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.985213] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1623.320881] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623482 with bad export cookie 14506286968428781666 [ 1623.326759] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.328361] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1623.721403] Lustre: server umount lustre-MDT0001 complete [ 1632.218292] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1634.674225] Key type lgssc unregistered [ 1634.915273] LNet: 55316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.927758] LNetError: 55316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.939838] LNet: Removed LNI 192.168.202.156@tcp [ 1635.747133] Key type .llcrypt unregistered [ 1635.748308] Key type ._llcrypt unregistered [ 1673.080460] Key type ._llcrypt registered [ 1673.082694] Key type .llcrypt registered [ 1673.178452] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1674.072472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1674.116752] alg: No test for adler32 (adler32-zlib) [ 1675.164140] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1675.381352] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1677.056270] Key type lgssc registered [ 1677.864987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1684.639911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1684.678903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.906403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1685.941953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1686.142983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.370798] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1698.181949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.285507] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1698.347923] Lustre: 57518:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1698.622953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.198620] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1709.220913] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.116466] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1720.383529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.631477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1720.888091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.157934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1727.132193] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1733.656566] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1739.255354] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1611h26m37s [ 1740.695695] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1747.699773] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1750.131851] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1750.182236] LustreError: 57526:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1770.661489] Lustre: lustre-MDT0000: Client 6788e700-2cfc-4dcd-bb83-742c1618a385 (at 192.168.202.56@tcp) reconnecting [ 1790.255109] LustreError: 57526:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1792.328928] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1794.588642] Lustre: DEBUG MARKER: done [ 1802.720079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1802.729730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.744556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.234863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.241345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.392480] Lustre: server umount lustre-OST0000 complete [ 1811.433258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1811.444910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.466847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.471108] Lustre: Skipped 2 previous similar messages [ 1812.516286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.519478] Lustre: Skipped 2 previous similar messages [ 1814.567251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1814.576387] Lustre: Skipped 1 previous similar message [ 1814.645423] Lustre: server umount lustre-MDT0000 complete [ 1817.781697] LustreError: 57512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623676 with bad export cookie 4616554677771518557 [ 1817.782845] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.787117] LustreError: 57512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.112330] Lustre: server umount lustre-MDT0001 complete [ 1828.729683] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1831.261250] Key type lgssc unregistered [ 1831.554632] LNet: 61845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.562756] LNetError: 61845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.578832] LNet: Removed LNI 192.168.202.156@tcp [ 1832.305605] Key type .llcrypt unregistered [ 1832.308881] Key type ._llcrypt unregistered [ 1869.992617] Key type ._llcrypt registered [ 1869.994770] Key type .llcrypt registered [ 1870.141178] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1870.994389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.023995] alg: No test for adler32 (adler32-zlib) [ 1872.039521] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 1872.205915] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1873.847194] Key type lgssc registered [ 1874.781682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.437892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.451451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.664549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1878.679925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1878.904662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.814542] Lustre: Failing over lustre-MDT0000 [ 1879.976180] Lustre: server umount lustre-MDT0000 complete [ 1884.939971] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:29:03 (1781623743) [ 1890.552842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.950504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.548379] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1901.348580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.413382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1901.438052] Lustre: 64622:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1901.646904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.705662] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1912.619551] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1917.197344] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1923.510581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.700625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1923.891230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.535391] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1933.435814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1935.002676] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1940.514574] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1945.926719] Lustre: server umount lustre-MDT0000 complete [ 1947.626443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1947.635882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.161474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.174872] Lustre: Skipped 1 previous similar message [ 1949.488166] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623808 with bad export cookie 1688294830258070228 [ 1949.498769] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.739964] Lustre: server umount lustre-MDT0001 complete [ 1952.380411] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1956.624401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1963.657979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.655479] Lustre: 63544:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623813/real 1781623813] req@0000000003c36727 x1868167885393536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1970.679498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.075215] LustreError: 66471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.087731] LustreError: 66471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1975.160944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.261431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1976.233697] Lustre: 63544:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623818/real 1781623818] req@0000000094830584 x1868167885393792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1978.542182] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1980.258691] LustreError: 68392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.259762] Lustre: 63546:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623823/real 1781623823] req@000000000a59baea x1868167885393920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.276372] LustreError: 68392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1985.439219] Lustre: 63546:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623828/real 1781623828] req@00000000e82e9f59 x1868167885394176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.506061] LustreError: 66472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.521117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.523365] LustreError: 66472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1985.775418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.451257] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1989.867627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1996.453881] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2001.789085] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2009.850506] Lustre: server umount lustre-MDT0000 complete [ 2010.612688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.637803] Lustre: Skipped 2 previous similar messages [ 2010.641783] LustreError: 69637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.668796] LustreError: 69637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2012.854621] LustreError: 65367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623871 with bad export cookie 1688294830258072510 [ 2012.855650] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.869944] LustreError: 65367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2013.054092] Lustre: server umount lustre-MDT0001 complete [ 2018.812739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2027.591930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.073470] Lustre: 63546:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623874/real 1781623874] req@00000000e9e51994 x1868167885417856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.081851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.225281] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x176e073401a93121 [ 2037.243554] Lustre: MGC192.168.202.156@tcp: Connection restored to 0@lo (at 0@lo) [ 2037.249617] Lustre: Skipped 1 previous similar message [ 2037.512707] LustreError: 70015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.582903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.628766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2041.736211] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2044.767400] Lustre: 63547:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781623887/real 1781623887] req@000000008a9f8783 x1868167885418496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781623903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2044.792052] Lustre: 63547:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2046.958472] LustreError: 71073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.984312] LustreError: 71073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2049.245196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.674676] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2053.127645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2059.041946] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.218063] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2069.986443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2069.997314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.012549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.536302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2073.855360] Lustre: server umount lustre-MDT0000 complete [ 2076.635793] LustreError: 65367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781623935 with bad export cookie 1688294830258073889 [ 2076.637328] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.641267] LustreError: 65367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2076.652149] LustreError: 71072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.657227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2076.660471] LustreError: 71072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2076.662056] Lustre: Skipped 2 previous similar messages [ 2076.839134] Lustre: server umount lustre-MDT0001 complete [ 2081.952955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2084.638981] Lustre: DEBUG MARKER: files compared the same [ 2086.342689] Lustre: server umount lustre-OST0000 complete [ 2099.264122] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2101.152735] Key type lgssc unregistered [ 2101.350442] LNet: 74343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.356195] LNetError: 74343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.375449] LNet: Removed LNI 192.168.202.156@tcp [ 2103.007772] Key type .llcrypt unregistered [ 2103.009751] Key type ._llcrypt unregistered [ 2121.738858] Key type ._llcrypt registered [ 2121.740845] Key type .llcrypt registered [ 2121.829715] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2122.690709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.731322] alg: No test for adler32 (adler32-zlib) [ 2123.756192] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2123.915759] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2125.567932] Key type lgssc registered [ 2126.361746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.345965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.367450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.725225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.266250] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2143.201841] LustreError: 75739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.211374] LustreError: 75739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2144.114032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.365967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.473535] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2153.330830] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2158.075483] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.741956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.985160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.138920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2175.336578] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2178.358777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2188.772222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.774255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.775243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.794214] Lustre: Skipped 1 previous similar message [ 2191.984909] Lustre: server umount lustre-OST0000 complete [ 2195.427098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2195.432091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.447572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.452981] Lustre: Skipped 1 previous similar message [ 2199.010316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.014652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.030881] Lustre: Skipped 1 previous similar message [ 2200.953761] Lustre: server umount lustre-MDT0000 complete [ 2203.775919] LustreError: 75719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624062 with bad export cookie 5098251069717776782 [ 2203.777053] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.786227] LustreError: 75719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2203.961663] Lustre: server umount lustre-MDT0001 complete [ 2212.992111] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2215.294558] Key type lgssc unregistered [ 2215.507502] LNet: 79271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.510932] LNetError: 79271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.520826] LNet: Removed LNI 192.168.202.156@tcp [ 2216.071963] Key type .llcrypt unregistered [ 2216.074147] Key type ._llcrypt unregistered [ 2236.353241] Key type ._llcrypt registered [ 2236.357432] Key type .llcrypt registered [ 2236.470964] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2237.570369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.662207] alg: No test for adler32 (adler32-zlib) [ 2238.708290] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2238.931395] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2240.615211] Key type lgssc registered [ 2241.692843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2249.187859] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2249.205650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.736502] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2269.855453] LustreError: 80642:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 2296.543254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.101168] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2308.465518] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2315.060808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.145274] Lustre: *** cfs_fail_loc=706, val=0*** [ 2315.424587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.486648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2318.714167] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2320.866169] LustreError: 80667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.877494] LustreError: 80667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2324.899413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.180884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.384194] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2333.915913] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2338.510305] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2340.833829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2340.844162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.858383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.346578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.351357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.367692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.464326] Lustre: server umount lustre-OST0000 complete [ 2351.072977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2351.081070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.090062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.097320] Lustre: Skipped 1 previous similar message [ 2355.994770] Lustre: server umount lustre-MDT0000 complete [ 2357.218315] LustreError: 81732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.953650] LustreError: 80649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624218 with bad export cookie 12146687067605921329 [ 2358.965632] LustreError: 80649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2358.965968] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.289168] Lustre: server umount lustre-MDT0001 complete [ 2368.736930] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2371.650786] Key type lgssc unregistered [ 2371.922993] LNet: 84253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.933535] LNetError: 84253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.963464] LNet: Removed LNI 192.168.202.156@tcp [ 2372.760947] Key type .llcrypt unregistered [ 2372.764898] Key type ._llcrypt unregistered [ 2393.026747] Key type ._llcrypt registered [ 2393.032128] Key type .llcrypt registered [ 2393.129600] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2393.992672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.021906] alg: No test for adler32 (adler32-zlib) [ 2394.992571] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2395.165563] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2396.816153] Key type lgssc registered [ 2397.867441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2404.093536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2404.116082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.709651] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2414.677829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.020498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2419.457504] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2420.194327] LustreError: 86336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.211573] LustreError: 86336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2425.315690] LustreError: 86340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.337957] LustreError: 86340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2425.784357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2425.971042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.189337] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2431.467554] LustreError: 86339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.483474] LustreError: 86339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2436.411653] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.418754] Lustre: MGS: Client 1397e530-22fa-412d-b80d-ba8458585a3c (at 0@lo) reconnecting [ 2436.434934] LustreError: 86350:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2436.597202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.656389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2440.414194] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2456.034934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.035529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.038472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.042778] Lustre: Skipped 1 previous similar message [ 2458.405466] Lustre: server umount lustre-OST0000 complete [ 2466.275117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.278988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.287567] Lustre: Skipped 1 previous similar message [ 2466.305732] Lustre: Skipped 2 previous similar messages [ 2467.489898] Lustre: server umount lustre-MDT0000 complete [ 2470.520563] Lustre: server umount lustre-MDT0001 complete [ 2473.761142] Lustre: server umount MGS complete [ 2482.222637] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:39:00 (1781624340) [ 2502.347905] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2509.279116] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2539.267727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.458780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2550.048667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.372640] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2550.427508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2550.533527] Lustre: lustre-MDT0001: new disk, initializing [ 2550.592846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2550.605303] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2550.609153] Lustre: 93500:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2550.613722] LustreError: 93500:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2554.030254] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2562.372792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.564275] Lustre: lustre-OST0000: new disk, initializing [ 2562.567578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2562.573659] Lustre: 94266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2562.638166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.770323] Lustre: 94352:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008e61e8e7 x1868168433799808/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781624439 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2567.600614] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2575.096572] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2580.449526] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.459596] Lustre: MGS: Client df6d37ff-58cc-4d7e-952b-53ada3bd1a13 (at 0@lo) reconnecting [ 2580.493917] Lustre: 95311:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2580.514827] LustreError: 93486:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2580.526112] LustreError: 93486:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2580.549955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2580.600416] Lustre: lustre-MDT0000: new disk, initializing [ 2580.699198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.719783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2580.746940] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2581.796090] Lustre: 95335:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000cc83a8e3 x1868168433806848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781624456 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2583.805694] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2586.914188] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2590.891535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2590.900859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2590.968901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2611.681871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.691329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.702037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.708582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.271768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.277431] Lustre: Skipped 2 previous similar messages [ 2616.756344] Lustre: server umount lustre-OST0000 complete [ 2620.384333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.385062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.388913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.392639] Lustre: Skipped 2 previous similar messages [ 2630.624470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.628650] Lustre: Skipped 3 previous similar messages [ 2633.183117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2633.269447] Lustre: server umount lustre-MDT0000 complete [ 2635.744844] LustreError: 93472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.769437] LustreError: 93472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2650.082309] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2650.341576] Lustre: server umount lustre-MDT0001 complete [ 2653.064953] Lustre: server umount MGS complete [ 2661.574642] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:42:00 (1781624520) [ 2674.323502] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2676.137989] Key type lgssc unregistered [ 2676.385550] LNet: 97831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.392814] LNetError: 97831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.420946] LNet: Removed LNI 192.168.202.156@tcp [ 2676.917323] Key type .llcrypt unregistered [ 2676.920551] Key type ._llcrypt unregistered [ 2688.641200] Key type ._llcrypt registered [ 2688.643346] Key type .llcrypt registered [ 2688.728636] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2689.454837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.502071] alg: No test for adler32 (adler32-zlib) [ 2690.474609] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2690.622287] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2692.255209] Key type lgssc registered [ 2693.189900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.743499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2698.770906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.119505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.288218] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2709.314492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.709947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.829078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2717.412933] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2721.524420] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2726.847255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.137151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.570878] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2737.572264] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2745.576827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2756.065649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2756.067375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.074220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.082352] Lustre: Skipped 1 previous similar message [ 2758.765531] Lustre: server umount lustre-OST0000 complete [ 2766.307339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.315579] LustreError: Skipped 1 previous similar message [ 2766.321369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.324833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.324841] Lustre: Skipped 1 previous similar message [ 2766.355692] Lustre: Skipped 1 previous similar message [ 2768.402659] Lustre: server umount lustre-MDT0000 complete [ 2771.414615] LustreError: 99011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624630 with bad export cookie 16382295700955013296 [ 2771.423267] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.424756] LustreError: 99011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2771.730294] Lustre: server umount lustre-MDT0001 complete [ 2779.331525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2781.654588] Key type lgssc unregistered [ 2781.885595] LNet: 102576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.891468] LNetError: 102576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.903747] LNet: Removed LNI 192.168.202.156@tcp [ 2782.682460] Key type .llcrypt unregistered [ 2782.684689] Key type ._llcrypt unregistered [ 2792.217291] Key type ._llcrypt registered [ 2792.219125] Key type .llcrypt registered [ 2792.292030] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2792.895230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.940913] alg: No test for adler32 (adler32-zlib) [ 2793.911452] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2794.020141] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2795.615332] Key type lgssc registered [ 2796.245715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2800.027604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2800.031572] LustreError: 103651:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2800.056504] LustreError: 103655:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2800.063451] LustreError: 103651:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 2800.071969] LustreError: 103651:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2800.076831] LustreError: 103651:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2800.087077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.436174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.263607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.449767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.362108] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2812.791412] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lsmod [ 2814.657729] LustreError: 104863:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2814.657771] LustreError: 104862:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2814.663234] LustreError: 104863:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2814.667163] LustreError: 104863:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2814.671601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.676531] LustreError: 104863:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2814.953542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.993573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2817.002490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.017138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.091037] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.091934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.096569] Lustre: Skipped 1 previous similar message [ 2821.100646] Lustre: Skipped 1 previous similar message [ 2826.220049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.223105] Lustre: Skipped 1 previous similar message [ 2831.329559] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2831.336352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.341986] Lustre: Skipped 2 previous similar messages [ 2831.428792] Lustre: server umount lustre-MDT0000 complete [ 2833.769852] LustreError: 103669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624692 with bad export cookie 15439925953340173678 [ 2833.775781] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.785787] LustreError: 103669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2833.978136] Lustre: server umount lustre-MDT0001 complete [ 2846.794913] Lustre: server umount lustre-OST0000 complete [ 2852.279257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.583824] LustreError: 105927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.596821] LustreError: 105927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2852.624853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.629776] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2857.954366] LustreError: 105928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.542136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.758110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.918576] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2869.522130] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2872.668729] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2876.410208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2876.594619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.610408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2880.117834] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2884.376602] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2893.282068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.293448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.305545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.377176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.366456] Lustre: server umount lustre-OST0000 complete [ 2902.497252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.503556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.503749] Lustre: Skipped 1 previous similar message [ 2902.512308] Lustre: Skipped 3 previous similar messages [ 2907.787835] Lustre: server umount lustre-MDT0000 complete [ 2909.913476] LustreError: 105907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624769 with bad export cookie 15439925953340174378 [ 2909.917257] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.918291] LustreError: 105907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.121316] Lustre: server umount lustre-MDT0001 complete [ 2916.685314] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2918.155691] Key type lgssc unregistered [ 2918.362436] LNet: 109457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2918.370762] LNetError: 109457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2918.387440] LNet: Removed LNI 192.168.202.156@tcp [ 2918.905592] Key type .llcrypt unregistered [ 2918.907658] Key type ._llcrypt unregistered [ 2933.658881] Key type ._llcrypt registered [ 2933.659855] Key type .llcrypt registered [ 2933.710888] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2934.331038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2934.369653] alg: No test for adler32 (adler32-zlib) [ 2935.332749] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 2935.517112] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2937.183142] Key type lgssc registered [ 2938.040638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.466435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2943.476426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.848619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.685562] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2953.544172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2953.745725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.908347] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2962.134788] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2965.462975] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2969.538381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.773132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.238401] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2978.598316] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.230844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2998.539089] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3003.433324] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3009.004025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.013941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.054966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.016790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.022669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.038313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.059044] Lustre: Skipped 1 previous similar message [ 3014.233258] Lustre: server umount lustre-OST0000 complete [ 3020.258316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.286825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.299375] Lustre: Skipped 1 previous similar message [ 3025.377701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.897950] Lustre: server umount lustre-MDT0000 complete [ 3030.508079] LustreError: 110847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.539339] LustreError: 110847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3031.600420] LustreError: 110832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781624890 with bad export cookie 5318883279600248350 [ 3031.614064] LustreError: 110832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3031.615076] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3032.265983] Lustre: server umount lustre-MDT0001 complete [ 3050.984950] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3055.145197] Key type lgssc unregistered [ 3055.549207] LNet: 114512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3055.566110] LNetError: 114512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3055.599431] LNet: Removed LNI 192.168.202.156@tcp [ 3057.168512] Key type .llcrypt unregistered [ 3057.170064] Key type ._llcrypt unregistered [ 3076.245995] Key type ._llcrypt registered [ 3076.247895] Key type .llcrypt registered [ 3076.337928] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3077.237139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.333066] alg: No test for adler32 (adler32-zlib) [ 3078.416187] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 3078.571025] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3080.263174] Key type lgssc registered [ 3081.270884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3088.035455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3088.053277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.683381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.585877] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3099.106602] LustreError: 115735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.126911] LustreError: 115735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3103.399621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.727442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3107.398118] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3114.411412] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.193152] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3126.682932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.995375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.491249] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3134.225837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3140.810166] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3148.769702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.774424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.783546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.799192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.801826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.805775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.822321] Lustre: Skipped 1 previous similar message [ 3153.926734] Lustre: server umount lustre-OST0000 complete [ 3160.032587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3160.033244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.041693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.059771] Lustre: Skipped 1 previous similar message [ 3160.082730] Lustre: Skipped 1 previous similar message [ 3163.961237] Lustre: server umount lustre-MDT0000 complete [ 3165.154344] LustreError: 115735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.455564] LustreError: 115716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625026 with bad export cookie 15952606041661703018 [ 3167.471342] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.808276] Lustre: server umount lustre-MDT0001 complete [ 3178.565956] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3181.269432] Key type lgssc unregistered [ 3181.539746] LNet: 119271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.550475] LNetError: 119271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3182.567820] LNet: Removed LNI 192.168.202.156@tcp [ 3183.413719] Key type .llcrypt unregistered [ 3183.415428] Key type ._llcrypt unregistered [ 3203.475369] Key type ._llcrypt registered [ 3203.478363] Key type .llcrypt registered [ 3203.615392] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3204.743271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.756299] alg: No test for adler32 (adler32-zlib) [ 3205.916362] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 3206.153776] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3207.849132] Key type lgssc registered [ 3208.994813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.914834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3215.939810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.319283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.329687] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3227.616694] LustreError: 120670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.634295] LustreError: 120670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3228.091389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.408845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.571358] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3239.920509] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3245.434667] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3251.591874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.835747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.977217] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3262.080130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3265.109549] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3271.300774] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3274.533783] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3274.538328] Lustre: Skipped 2 previous similar messages [ 3276.858126] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3276.873988] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3278.590902] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3278.596185] Lustre: Skipped 2 previous similar messages [ 3284.532515] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3284.536285] Lustre: Skipped 1 previous similar message [ 3288.542783] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3288.548603] Lustre: Skipped 2 previous similar messages [ 3291.753719] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 3297.714337] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.56/24@tcp in log lustre-MDT0000 [ 3297.720436] Lustre: Skipped 2 previous similar messages [ 3299.544616] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 3299.554555] Lustre: Skipped 1 previous similar message [ 3308.013581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3308.013737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.018141] LustreError: Skipped 2 previous similar messages [ 3308.037805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.522588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.641957] Lustre: server umount lustre-OST0000 complete [ 3318.759731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.780052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3318.785089] Lustre: Skipped 2 previous similar messages [ 3322.481256] Lustre: server umount lustre-MDT0000 complete [ 3323.888727] LustreError: 120665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.912817] LustreError: 120665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3325.660304] LustreError: 120651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625184 with bad export cookie 3134652262746668748 [ 3325.675223] LustreError: 120651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3325.675818] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.032306] Lustre: server umount lustre-MDT0001 complete [ 3335.742918] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3338.099941] Key type lgssc unregistered [ 3338.381770] LNet: 125516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.386328] LNetError: 125516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.400299] LNet: Removed LNI 192.168.202.156@tcp [ 3339.047341] Key type .llcrypt unregistered [ 3339.050887] Key type ._llcrypt unregistered [ 3364.630469] Key type ._llcrypt registered [ 3364.632672] Key type .llcrypt registered [ 3364.760164] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3366.087442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3366.163352] alg: No test for adler32 (adler32-zlib) [ 3367.284977] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 3367.659651] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3369.335186] Key type lgssc registered [ 3370.623043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3378.530865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3378.580658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.382299] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3380.403790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.411690] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.56/24@tcp [ 3384.734506] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3389.925941] LustreError: 127057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.936960] LustreError: 127057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3392.803846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.129285] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3393.152294] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 3393.219248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.270855] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3403.512443] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3408.090878] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3413.944990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.284976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.779335] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3425.580051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3425.666708] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3436.004565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.011717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.011881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.016526] Lustre: Skipped 1 previous similar message [ 3440.321543] Lustre: server umount lustre-OST0000 complete [ 3444.704852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3444.706893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.718876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.728269] Lustre: Skipped 1 previous similar message [ 3446.240482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3446.247085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.465333] Lustre: server umount lustre-MDT0000 complete [ 3451.368035] LustreError: 127054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.384364] LustreError: 127054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3452.042865] LustreError: 127797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625311 with bad export cookie 2207217289941604159 [ 3452.046197] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.048342] LustreError: 127797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3452.296825] Lustre: server umount lustre-MDT0001 complete [ 3460.410582] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3462.824837] Key type lgssc unregistered [ 3463.009506] LNet: 130642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.012669] LNetError: 130642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3463.027945] LNet: Removed LNI 192.168.202.156@tcp [ 3463.626179] Key type .llcrypt unregistered [ 3463.628669] Key type ._llcrypt unregistered [ 3483.481324] Key type ._llcrypt registered [ 3483.482670] Key type .llcrypt registered [ 3483.543687] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3495.605121] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3496.886931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.912667] alg: No test for adler32 (adler32-zlib) [ 3497.984822] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 3498.259572] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3499.896279] Key type lgssc registered [ 3500.910772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3531.581661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3531.672330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.878428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3532.908474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3532.961741] Lustre: lustre-MDT0000: new disk, initializing [ 3533.039778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.058302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3537.186729] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3546.349466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.516567] Lustre: 134020:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3546.579863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3546.583452] Lustre: Skipped 1 previous similar message [ 3546.631699] Lustre: lustre-MDT0001: new disk, initializing [ 3546.698354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3546.728297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3546.745615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3550.235672] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3558.352107] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3563.488502] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3568.909251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.111696] Lustre: lustre-OST0000: new disk, initializing [ 3569.114686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3569.119827] Lustre: 135987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3569.195983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.980615] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3574.820840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3574.837277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3574.928506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3581.698831] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.500987] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3585.692983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3589.050527] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3589.230659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3595.239657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3595.242847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.265197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.749556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.772581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.868588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.871124] Lustre: Skipped 1 previous similar message [ 3604.962249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3605.084142] Lustre: server umount lustre-OST0000 complete [ 3608.042683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3608.048417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.065391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.107219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.224754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.229248] Lustre: Skipped 3 previous similar messages [ 3622.367160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3622.555811] Lustre: server umount lustre-MDT0000 complete [ 3625.657216] LustreError: 134014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625484 with bad export cookie 15754153539613822046 [ 3625.669595] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.672084] LustreError: 134014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3625.913236] Lustre: server umount lustre-MDT0001 complete [ 3632.704554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.234395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.173930] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3638.244483] LustreError: 138206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.258248] LustreError: 138206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3643.360230] LustreError: 138205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.923665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.193841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3647.916433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3654.015610] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3657.969826] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3663.829823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.030781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.500673] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3676.005403] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3679.296655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3679.476521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3682.797519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3682.972821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3689.839529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.952455] Lustre: lustre-OST0001: new disk, initializing [ 3689.954547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3689.957293] Lustre: 141574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3690.031656] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.250604] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3700.296451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3700.305709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3700.395809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3702.054288] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3702.270756] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3706.296345] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3706.601859] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3710.899324] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b6a42983000.ost_server_uuid 50 [ 3712.165805] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b6a42983000.ost_server_uuid in FULL state after 0 sec [ 3721.191371] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3721.195304] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.198756] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3721.201301] Lustre: Skipped 1 previous similar message [ 3721.206433] Lustre: Skipped 2 previous similar messages [ 3723.471177] Lustre: server umount lustre-OST0001 complete [ 3726.313225] LustreError: 140025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.342657] LustreError: 140025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3731.427882] LustreError: 140026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.439078] LustreError: 140026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3732.149479] Lustre: server umount lustre-OST0000 complete [ 3736.553263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3736.557494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.559159] LustreError: Skipped 1 previous similar message [ 3736.571097] Lustre: Skipped 3 previous similar messages [ 3741.664695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.671778] Lustre: Skipped 7 previous similar messages [ 3749.343178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.454246] Lustre: server umount lustre-MDT0000 complete [ 3751.905031] LustreError: 139272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.913968] LustreError: 139272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3751.969552] LustreError: 141573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625611 with bad export cookie 15754153539613822928 [ 3751.971496] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.979793] LustreError: 141573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3752.286345] Lustre: server umount lustre-MDT0001 complete [ 3765.151464] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3767.043936] Key type lgssc unregistered [ 3767.280860] LNet: 144577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3767.287671] LNetError: 144577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.295101] LNet: Removed LNI 192.168.202.156@tcp [ 3768.994458] Key type .llcrypt unregistered [ 3768.996058] Key type ._llcrypt unregistered [ 3797.027689] Key type ._llcrypt registered [ 3797.028708] Key type .llcrypt registered [ 3797.101326] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3806.987631] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3807.994331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3808.009476] alg: No test for adler32 (adler32-zlib) [ 3808.989779] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 3809.185446] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3810.863232] Key type lgssc registered [ 3811.821168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3837.119390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3837.145241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.381842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3838.410682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3838.473864] Lustre: lustre-MDT0000: new disk, initializing [ 3838.545204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.555186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3841.888858] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3849.476503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.548091] Lustre: 148718:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3849.584599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3849.589611] Lustre: Skipped 1 previous similar message [ 3849.646124] Lustre: lustre-MDT0001: new disk, initializing [ 3849.733944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3849.761995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3849.769259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3852.740377] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3859.312548] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3863.412632] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3868.483547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3868.680647] Lustre: lustre-OST0000: new disk, initializing [ 3868.683149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3868.686919] Lustre: 150689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3868.763524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.129601] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3874.842416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.847758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.930297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3880.325753] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3884.385981] Lustre: Failing over lustre-OST0000 [ 3884.536996] Lustre: server umount lustre-OST0000 complete [ 3901.277574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.407146] Lustre: 146943:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625744/real 1781625744] req@000000003ddd2556 x1868169916479104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781625760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3901.428676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.566153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.579825] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3902.623074] Lustre: 146946:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625744/real 1781625744] req@0000000074a0d833 x1868169916479232/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781625760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3903.157715] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3905.619762] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3906.719117] Lustre: 146945:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625749/real 1781625749] req@000000003bbeb762 x1868169916481024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781625765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3906.740829] Lustre: 146945:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3907.069322] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 3907.071071] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3908.340083] Lustre: Failing over lustre-MDT0000 [ 3908.558772] Lustre: server umount lustre-MDT0000 complete [ 3911.140687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3911.153648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.170615] Lustre: Skipped 1 previous similar message [ 3911.180081] LustreError: 150403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.201064] LustreError: 150403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3911.393028] Lustre: 146946:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625754/real 1781625754] req@000000000212bdaa x1868169916482176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781625770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3912.163236] LustreError: 148732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.165778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.174536] LustreError: 148732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3912.193444] Lustre: Skipped 1 previous similar message [ 3913.403500] LustreError: 148727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.411678] LustreError: 148727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3916.256047] Lustre: 146943:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781625759/real 1781625759] req@000000005121b380 x1868169916483328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781625775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3917.285812] LustreError: 148731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.302734] LustreError: 148731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3922.403949] LustreError: 148728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.415367] LustreError: 148728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3925.024024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.106930] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3925.301606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.364613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3928.929422] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3929.749981] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3930.605931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3930.610153] Lustre: Skipped 1 previous similar message [ 3930.636526] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3930.669470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3937.761149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.766534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.779187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.844081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.956571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.965837] Lustre: Skipped 2 previous similar messages [ 3951.071700] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3951.074741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.080219] Lustre: Skipped 1 previous similar message [ 3951.183319] Lustre: server umount lustre-OST0000 complete [ 3956.192087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3956.193121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.197666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.215429] Lustre: Skipped 2 previous similar messages [ 3956.225729] Lustre: Skipped 1 previous similar message [ 3959.969119] Lustre: server umount lustre-MDT0000 complete [ 3961.314590] LustreError: 150137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.331924] LustreError: 150137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3963.007089] LustreError: 148711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781625822 with bad export cookie 11790479840736759794 [ 3963.008335] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.015890] LustreError: 148711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3963.261586] Lustre: server umount lustre-MDT0001 complete [ 3971.001525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3973.177254] Key type lgssc unregistered [ 3973.417501] LNet: 154337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3973.421868] LNetError: 154337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3973.433151] LNet: Removed LNI 192.168.202.156@tcp [ 3974.079540] Key type .llcrypt unregistered [ 3974.082239] Key type ._llcrypt unregistered [ 4002.140872] Key type ._llcrypt registered [ 4002.142790] Key type .llcrypt registered [ 4002.198577] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4002.868209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4002.877483] alg: No test for adler32 (adler32-zlib) [ 4003.800270] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 4003.912674] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4005.511508] Key type lgssc registered [ 4006.177247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4010.982550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4010.993142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.250252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.003181] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4021.074570] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4022.753033] LustreError: 155855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.761372] LustreError: 155855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4023.854830] LustreError: 155884:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4023.861549] LustreError: 155884:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4023.864368] LustreError: 155884:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4024.081316] Lustre: server umount lustre-MDT0000 complete [ 4032.893648] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4038.725239] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4060.223398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.384983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.413368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.472853] Lustre: lustre-MDT0000: new disk, initializing [ 4060.530849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.553561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4063.148833] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4070.512720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.589556] Lustre: 159185:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4070.628285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4070.631099] Lustre: Skipped 1 previous similar message [ 4070.686609] Lustre: lustre-MDT0001: new disk, initializing [ 4070.754901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.786971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4070.795716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4073.771714] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4080.462872] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4084.065961] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4088.867070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.019597] Lustre: lustre-OST0000: new disk, initializing [ 4089.022228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4089.025113] Lustre: 161154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4089.076270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.570443] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4094.997613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4095.002929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4095.079705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4098.598892] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4102.480761] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4103.506782] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:06:02 (1781625962) [ 4263.903853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.904277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.909939] Lustre: Skipped 2 previous similar messages [ 4263.912721] Lustre: Skipped 1 previous similar message [ 4269.023703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.027716] Lustre: Skipped 3 previous similar messages [ 4269.869367] Lustre: server umount lustre-MDT0000 complete [ 4272.024485] LustreError: 161153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626131 with bad export cookie 17729754694569645895 [ 4272.028089] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4272.032842] LustreError: 161153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4274.143805] LustreError: 159191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.144936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.161104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4277.151968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4277.152032] LustreError: 159192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.282092] Lustre: server umount lustre-MDT0001 complete [ 4286.501931] Lustre: server umount lustre-OST0000 complete [ 4292.317735] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4299.547370] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4325.721820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.002351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4326.050656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4326.130931] Lustre: lustre-MDT0000: new disk, initializing [ 4326.198325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.213223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4329.224295] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4336.561819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.665883] Lustre: 165802:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4336.698958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4336.701577] Lustre: Skipped 1 previous similar message [ 4336.755346] Lustre: lustre-MDT0001: new disk, initializing [ 4336.813597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.837434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4336.846474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4339.288774] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4345.433210] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4348.634418] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4352.608912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4352.754266] Lustre: lustre-OST0000: new disk, initializing [ 4352.756658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4352.760709] Lustre: 167766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4355.997243] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4360.732732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4360.743271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4360.822354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4362.286503] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4365.336592] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4365.462981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4368.576953] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4368.720399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4370.912299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4370.917398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.928317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.932512] Lustre: Skipped 1 previous similar message [ 4381.156316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.160843] Lustre: Skipped 3 previous similar messages [ 4384.223822] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4384.344402] Lustre: server umount lustre-OST0000 complete [ 4388.320801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4388.325802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.338725] Lustre: Skipped 1 previous similar message [ 4392.902662] Lustre: server umount lustre-MDT0000 complete [ 4395.225931] LustreError: 165793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626254 with bad export cookie 17729754694569804732 [ 4395.226642] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.234478] LustreError: 165793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4395.444502] Lustre: server umount lustre-MDT0001 complete [ 4400.491834] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:10:59 (1781626259) [ 4410.579050] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4416.168995] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4439.415337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.588112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4439.610480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4439.666753] Lustre: lustre-MDT0000: new disk, initializing [ 4439.716239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.718828] Lustre: Skipped 1 previous similar message [ 4439.731866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4442.762201] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4450.890381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.011364] Lustre: 172524:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4451.220220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4453.967703] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4460.600464] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4464.078137] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4468.811997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.018806] Lustre: 174499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4473.075882] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4478.000850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4478.006254] Lustre: Skipped 1 previous similar message [ 4478.014176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4478.082619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4479.477386] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4488.160058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.164698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.174055] Lustre: Skipped 1 previous similar message [ 4488.177369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.180214] Lustre: Skipped 3 previous similar messages [ 4489.767610] Lustre: server umount lustre-OST0000 complete [ 4492.256913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4492.262412] LustreError: Skipped 1 previous similar message [ 4498.099785] Lustre: server umount lustre-MDT0000 complete [ 4498.911947] LustreError: 172534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.923767] LustreError: 172534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4500.908778] LustreError: 172515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626360 with bad export cookie 17729754694569805614 [ 4500.915205] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.917332] LustreError: 172515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4501.266313] Lustre: server umount lustre-MDT0001 complete [ 4507.207620] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:12:45 (1781626365) [ 4517.780677] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4523.828392] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4546.727230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.870693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4546.887626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4546.889899] Lustre: Skipped 3 previous similar messages [ 4546.939320] Lustre: lustre-MDT0000: new disk, initializing [ 4546.941594] Lustre: Skipped 2 previous similar messages [ 4546.990427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.994864] Lustre: Skipped 2 previous similar messages [ 4547.007089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4549.659030] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4557.047603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.120788] Lustre: 178811:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4557.298220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4560.007484] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4566.250768] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4569.679984] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4573.709633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.851216] Lustre: 180789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4577.225533] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4582.539379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4583.422423] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4588.005913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.011280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.019378] Lustre: Skipped 3 previous similar messages [ 4588.020753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.032370] Lustre: Skipped 4 previous similar messages [ 4593.874347] Lustre: server umount lustre-OST0000 complete [ 4598.240165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4598.245056] LustreError: Skipped 1 previous similar message [ 4602.619150] Lustre: server umount lustre-MDT0000 complete [ 4603.366465] LustreError: 178825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.181120] LustreError: 178805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626464 with bad export cookie 17729754694569806692 [ 4605.183233] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.197360] LustreError: 178805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4605.418526] Lustre: server umount lustre-MDT0001 complete [ 4613.025930] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4614.690394] Key type lgssc unregistered [ 4614.902761] LNet: 182642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4614.907207] LNetError: 182642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4614.928091] LNet: Removed LNI 192.168.202.156@tcp [ 4615.469108] Key type .llcrypt unregistered [ 4615.471991] Key type ._llcrypt unregistered [ 4630.315404] Key type ._llcrypt registered [ 4630.316653] Key type .llcrypt registered [ 4630.387074] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4631.134781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4631.147910] alg: No test for adler32 (adler32-zlib) [ 4632.121753] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 4632.292077] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4633.895514] Key type lgssc registered [ 4634.463598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4638.200626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4638.214264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.473184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.513444] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4645.934686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.153846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.403181] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4652.089426] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4655.180697] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4659.316900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.508429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.599984] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4664.624069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4666.259534] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4685.289575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4685.295664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.303837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.328204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.331194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.331390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4691.106869] Lustre: server umount lustre-OST0000 complete [ 4697.569344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.570791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.581538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.592807] Lustre: Skipped 2 previous similar messages [ 4702.692356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.298232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4707.402650] Lustre: server umount lustre-MDT0000 complete [ 4707.808362] LustreError: 184036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.816141] LustreError: 184036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4709.632973] LustreError: 184018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626568 with bad export cookie 5781627686109121870 [ 4709.633785] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.638576] LustreError: 184018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4709.819928] Lustre: server umount lustre-MDT0001 complete [ 4715.936875] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4717.532921] Key type lgssc unregistered [ 4717.694572] LNet: 187589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.698202] LNetError: 187589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4717.709197] LNet: Removed LNI 192.168.202.156@tcp [ 4718.099433] Key type .llcrypt unregistered [ 4718.101099] Key type ._llcrypt unregistered [ 4731.098719] Key type ._llcrypt registered [ 4731.100067] Key type .llcrypt registered [ 4731.158701] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4731.743217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4731.769227] alg: No test for adler32 (adler32-zlib) [ 4732.647283] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 4732.752124] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4734.359234] Key type lgssc registered [ 4734.964985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4739.493959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4739.502731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.761703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4743.369949] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4748.413918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.612570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.497962] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4754.462908] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4757.752488] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4761.890267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4762.115803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.230080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4765.765811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4769.983887] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4778.468324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.474880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4778.483064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.499510] Lustre: Skipped 1 previous similar message [ 4779.807520] Lustre: server umount lustre-OST0000 complete [ 4783.843519] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b6a45c7e000.ost_server_uuid 50 [ 4799.601492] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4813.280469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.281390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4813.284856] Lustre: Skipped 2 previous similar messages [ 4816.582792] Lustre: server umount lustre-MDT0000 complete [ 4818.396316] LustreError: 188965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626677 with bad export cookie 5531771336496120017 [ 4818.397134] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.402331] LustreError: 188965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4818.571950] Lustre: server umount lustre-MDT0001 complete [ 4822.862456] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:18:01 (1781626681) [ 4826.222859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.466178] LustreError: 192899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4826.476724] LustreError: 192899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4826.522368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.636246] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4831.712924] LustreError: 192900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.186743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.381405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4835.428229] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4839.299491] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4841.830517] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4844.896062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.060914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.709227] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4851.108510] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4851.245586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4854.323559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4854.431846] Lustre: lustre-OST0001: new disk, initializing [ 4854.434706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4854.438799] Lustre: 195736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4857.219114] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4857.886898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4857.891990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4857.928582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4861.992375] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4863.978786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.981808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.990392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4863.995507] Lustre: Skipped 1 previous similar message [ 4863.998956] Lustre: Skipped 1 previous similar message [ 4867.746933] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 4869.775362] Lustre: server umount lustre-OST0000 complete [ 4872.708893] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4872.809063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4872.850518] LustreError: 194720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4874.581382] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4874.668796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4877.963471] LustreError: 196182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.971262] LustreError: 196182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4883.092562] LustreError: 194720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.101787] LustreError: 194720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4888.545889] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4888.545981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.548316] LustreError: Skipped 1 previous similar message [ 4888.549697] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4888.549703] Lustre: Skipped 2 previous similar messages [ 4888.563724] Lustre: Skipped 1 previous similar message [ 4891.175664] Lustre: server umount lustre-OST0001 complete [ 4894.689455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4894.689623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.125436] Lustre: server umount lustre-MDT0000 complete [ 4899.807824] LustreError: 192899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.815956] LustreError: 192899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4900.874179] LustreError: 192879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626759 with bad export cookie 5531771336496121088 [ 4900.877788] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.880201] LustreError: 192879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.025813] Lustre: server umount lustre-MDT0001 complete [ 4912.160959] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:19:30 (1781626770) [ 4915.375079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.528938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4915.549275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4915.652871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.655652] Lustre: Skipped 1 previous similar message [ 4917.725234] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4922.648055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.741549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4922.768639] Lustre: 199149:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4925.012531] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4928.066735] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4930.455584] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4933.354941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.467164] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4936.213122] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4939.458267] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4942.418964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4942.470386] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4942.614514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4945.492978] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4949.155787] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4953.064810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.067714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.070154] LustreError: Skipped 1 previous similar message [ 4953.077917] Lustre: Skipped 1 previous similar message [ 4953.081038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.089103] Lustre: Skipped 6 previous similar messages [ 4956.592307] Lustre: server umount lustre-OST0000 complete [ 4958.180963] LustreError: 202055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.196824] LustreError: 202055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4968.416048] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4973.543727] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4973.550050] Lustre: Skipped 4 previous similar messages [ 4974.044317] Lustre: server umount lustre-OST0001 complete [ 4978.656668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4978.663836] Lustre: Skipped 4 previous similar messages [ 4981.949758] Lustre: server umount lustre-MDT0000 complete [ 4983.580377] LustreError: 201020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626842 with bad export cookie 5531771336496122488 [ 4983.581158] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4983.585104] LustreError: 201020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4983.739271] Lustre: server umount lustre-MDT0001 complete [ 4993.290259] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:20:52 (1781626852) [ 4999.508456] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5002.908397] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5016.002426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.102223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5016.114449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5016.154109] Lustre: lustre-MDT0000: new disk, initializing [ 5016.185935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.188615] Lustre: Skipped 3 previous similar messages [ 5016.197211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5017.686429] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5022.426417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.475238] Lustre: 207213:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5022.500247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5022.502259] Lustre: Skipped 1 previous similar message [ 5022.540387] Lustre: lustre-MDT0001: new disk, initializing [ 5022.590724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5022.594755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5024.047582] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5028.311464] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5030.566951] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5033.557935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.673929] Lustre: lustre-OST0000: new disk, initializing [ 5033.675756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5033.678937] Lustre: 209180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5036.195139] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5040.826574] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5041.154191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5041.157956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5041.203563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5042.796043] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5042.892445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5044.628732] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5044.722820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5046.242070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.242513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.244788] LustreError: Skipped 2 previous similar messages [ 5046.245716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.249301] Lustre: Skipped 1 previous similar message [ 5046.251281] Lustre: Skipped 2 previous similar messages [ 5047.637896] Lustre: server umount lustre-OST0000 complete [ 5055.376138] Lustre: server umount lustre-MDT0000 complete [ 5056.481301] LustreError: 208014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.487432] LustreError: 208014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5057.141654] LustreError: 207206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781626916 with bad export cookie 5531771336496123965 [ 5057.142081] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5057.148936] LustreError: 207206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5057.339842] Lustre: server umount lustre-MDT0001 complete [ 5061.395083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.304368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5067.352580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.770075] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5073.326723] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5076.162244] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.487574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.592569] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5085.732780] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5087.355492] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5087.436596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.167966] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5092.335119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 5097.441668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.445199] LustreError: Skipped 1 previous similar message [ 5099.235670] Lustre: server umount lustre-OST0000 complete [ 5101.527342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5101.609948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5103.201815] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5127.206856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5127.358080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.361015] Lustre: Skipped 5 previous similar messages [ 5129.952883] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5133.040907] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5133.784431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5137.313676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.316734] Lustre: Skipped 5 previous similar messages [ 5141.405099] Lustre: server umount lustre-OST0000 complete [ 5144.543843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.548611] Lustre: Skipped 4 previous similar messages [ 5149.380111] Lustre: server umount lustre-MDT0000 complete [ 5151.115447] LustreError: 211374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627010 with bad export cookie 5531771336496124833 [ 5151.118503] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5151.121643] LustreError: 211374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5151.273246] Lustre: server umount lustre-MDT0001 complete [ 5154.670020] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:23:33 (1781627013) [ 5157.705265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.927829] LustreError: 217391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.936375] LustreError: 217391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5159.761985] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5163.003303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.831187] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5167.870179] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5170.017884] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5172.619145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.131844] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5178.239717] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5180.009454] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5183.186422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5184.984534] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5185.073118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5187.750972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.823254] Lustre: lustre-OST0001: new disk, initializing [ 5187.825107] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5187.828039] Lustre: 220778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5190.179705] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5194.693196] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5194.763348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5194.767950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5194.822686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5196.361721] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5196.447172] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5198.152840] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5198.244422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5199.841926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.844607] LustreError: Skipped 2 previous similar messages [ 5201.088583] Lustre: server umount lustre-OST0001 complete [ 5203.542547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5203.639551] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5205.538214] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5205.634462] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5229.745316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5232.301432] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5235.177628] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5235.184512] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5235.184996] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5235.407604] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5236.153467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5243.205525] Lustre: server umount lustre-OST0001 complete [ 5265.889930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.891835] Lustre: Skipped 12 previous similar messages [ 5269.983157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5270.059163] Lustre: server umount lustre-OST0000 complete [ 5276.128518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.132720] Lustre: Skipped 8 previous similar messages [ 5286.367303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5286.445165] Lustre: server umount lustre-MDT0000 complete [ 5287.893447] LustreError: 217372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627147 with bad export cookie 5531771336496125764 [ 5287.897378] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.899447] LustreError: 217372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5302.239113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5302.337548] Lustre: server umount lustre-MDT0001 complete [ 5312.060098] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:26:10 (1781627170) [ 5314.591255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.710905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5314.727411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5314.813775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.816077] Lustre: Skipped 5 previous similar messages [ 5316.423895] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5320.049829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.103922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5320.117292] Lustre: 226013:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5322.140877] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5325.510977] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5328.322819] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5331.337128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5331.437258] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5334.066358] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5337.534950] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5347.426921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.470747] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5350.319800] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5353.502402] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5354.245744] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5356.323960] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5356.418660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5358.299591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5358.389785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5360.876440] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b6a47508000.ost_server_uuid 50 [ 5361.680571] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b6a47508000.ost_server_uuid in FULL state after 0 sec [ 5362.324347] Lustre: Permanently deactivating lustre-OST0001 [ 5362.336708] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5363.952390] Lustre: Permanently reactivating lustre-OST0001 [ 5368.808175] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.813031] LustreError: Skipped 6 previous similar messages [ 5372.048552] Lustre: server umount lustre-OST0001 complete [ 5373.921478] LustreError: 227855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.927303] LustreError: 227855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 5388.256055] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5388.328178] Lustre: server umount lustre-OST0000 complete [ 5396.077330] Lustre: server umount lustre-MDT0000 complete [ 5397.676307] LustreError: 226007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627256 with bad export cookie 5531771336496127038 [ 5397.680636] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5397.684866] LustreError: 226007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5397.818227] Lustre: server umount lustre-MDT0001 complete [ 5407.906476] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:27:46 (1781627266) [ 5411.150265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.261337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5411.275312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.279881] Lustre: Skipped 5 previous similar messages [ 5413.092939] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5416.535704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.595385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5416.614582] Lustre: 232698:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5418.406282] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5421.292572] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.352243] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.258805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.378226] Lustre: Permanently deactivating lustre-OST0000 [ 5426.385731] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5428.936519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5429.747738] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5429.753461] LustreError: 235019:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5429.756818] LustreError: 235019:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5431.820417] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5434.406164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5436.843286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5439.836323] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5440.052494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5445.709566] Lustre: Permanently reactivating lustre-OST0000 [ 5445.716134] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5445.718988] Lustre: Skipped 2 previous similar messages [ 5449.617264] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5449.621373] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.629027] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5449.632614] Lustre: Skipped 1 previous similar message [ 5449.646764] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5449.655154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5458.122374] Lustre: server umount lustre-OST0001 complete [ 5461.845585] Lustre: server umount lustre-OST0000 complete [ 5469.543615] Lustre: server umount lustre-MDT0000 complete [ 5470.841172] LustreError: 234579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627329 with bad export cookie 5531771336496128669 [ 5470.844065] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.846457] LustreError: 234579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5470.975203] Lustre: server umount lustre-MDT0001 complete [ 5475.100246] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 5476.230373] Key type lgssc unregistered [ 5476.369594] LNet: 237553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5476.371902] LNetError: 237553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5476.382323] LNet: Removed LNI 192.168.202.156@tcp [ 5476.733000] Key type .llcrypt unregistered [ 5476.734632] Key type ._llcrypt unregistered [ 5485.609614] Key type ._llcrypt registered [ 5485.611269] Key type .llcrypt registered [ 5485.664558] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5486.177699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5486.185174] alg: No test for adler32 (adler32-zlib) [ 5487.053246] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 5487.154875] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 5488.735196] Key type lgssc registered [ 5489.142559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5493.082497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5493.089779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.270360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.764154] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5499.155073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.212844] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5499.225950] Lustre: 238986:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5499.233079] Lustre: Permanently deactivating lustre-MDT0001 [ 5499.240413] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5499.355271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.831060] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5502.507422] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5502.510028] LustreError: 239026:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5502.518144] LustreError: 239026:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5502.524356] LustreError: 239026:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 5503.587841] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.230921] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5511.071633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.474282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.473820] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5519.660469] LustreError: 240834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.676202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5520.047684] LustreError: 240834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5523.803607] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5524.786654] LustreError: 241316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.806987] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5524.812881] LustreError: 241316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5529.833035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.057169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.210787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5535.217772] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5535.657765] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5543.242955] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5547.535858] Lustre: Permanently reactivating lustre-MDT0001 [ 5547.594837] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5547.598659] Lustre: Skipped 2 previous similar messages [ 5552.303995] Lustre: lustre-MDT0001: Client 7d0e0007-2861-4b15-9860-a2c51f797f25 (at 192.168.202.56@tcp) reconnecting [ 5557.667864] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5559.001784] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5563.667510] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5563.884480] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5567.886339] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5568.069393] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5570.221742] Lustre: Permanently deactivating lustre-MDT0001 [ 5570.232856] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5570.238377] Lustre: Skipped 2 previous similar messages [ 5578.269814] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5586.404964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5586.408856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.413968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5586.433930] Lustre: Skipped 1 previous similar message [ 5587.319814] Lustre: server umount lustre-MDT0000 complete [ 5589.619254] LustreError: 243476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627448 with bad export cookie 18123727783176458700 [ 5589.636672] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.849802] Lustre: server umount lustre-MDT0001 complete [ 5591.867479] Lustre: server umount lustre-OST0000 complete [ 5598.317227] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:30:56 (1781627456) [ 5603.548538] Lustre: server umount lustre-OST0001 complete [ 5606.939615] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5612.017024] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5633.795951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.963047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5633.970980] Lustre: Skipped 2 previous similar messages [ 5633.983754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5634.027952] Lustre: lustre-MDT0000: new disk, initializing [ 5634.090991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.107309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5636.748520] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5643.334569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.399808] Lustre: 247293:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5643.433425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5643.436594] Lustre: Skipped 1 previous similar message [ 5643.479183] Lustre: lustre-MDT0001: new disk, initializing [ 5643.536060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5643.559930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5643.568330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5645.758289] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5650.950616] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5654.053972] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5657.722389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5657.863207] Lustre: lustre-OST0000: new disk, initializing [ 5657.865428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5657.868968] Lustre: 249260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5660.591613] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5665.804379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5665.809724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5665.880969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5665.892401] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5667.176840] LustreError: 247301:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5673.092681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.194489] Lustre: lustre-OST0001: new disk, initializing [ 5673.197619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5673.200536] Lustre: 250372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5673.239212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5673.242347] Lustre: Skipped 1 previous similar message [ 5674.051598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5674.057235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5674.093871] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5676.916735] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5677.232049] LustreError: 247301:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5680.334728] LustreError: 247302:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5683.407840] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5684.460696] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5690.375144] LustreError: 247302:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5694.954220] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5694.957059] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.962669] Lustre: Skipped 2 previous similar messages [ 5694.966306] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5694.969620] Lustre: Skipped 4 previous similar messages [ 5698.080456] Lustre: server umount lustre-OST0001 complete [ 5700.077068] LustreError: 249242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.088717] LustreError: 249242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5705.185196] LustreError: 250463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.198616] LustreError: 250463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5710.312054] LustreError: 249245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.323219] LustreError: 249245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5715.430883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.436348] Lustre: Skipped 1 previous similar message [ 5715.439566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5715.443564] Lustre: Skipped 2 previous similar messages [ 5718.349608] Lustre: server umount lustre-OST0000 complete [ 5725.671195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.671311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5725.672213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.686227] Lustre: Skipped 2 previous similar messages [ 5725.689420] LustreError: Skipped 1 previous similar message [ 5730.786162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.788549] Lustre: Skipped 2 previous similar messages [ 5735.391110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5735.564759] Lustre: server umount lustre-MDT0000 complete [ 5735.906222] LustreError: 250224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.920525] LustreError: 250224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5737.810184] LustreError: 247287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627596 with bad export cookie 18123727783176461507 [ 5737.810570] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.815407] LustreError: 247287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5752.287212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5752.457601] Lustre: server umount lustre-MDT0001 complete [ 5760.260415] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 5762.156027] Key type lgssc unregistered [ 5762.368601] LNet: 252409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5762.373919] LNetError: 252409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5763.429463] LNet: Removed LNI 192.168.202.156@tcp [ 5764.052112] Key type .llcrypt unregistered [ 5764.053252] Key type ._llcrypt unregistered [ 5791.618446] Key type ._llcrypt registered [ 5791.620699] Key type .llcrypt registered [ 5791.688149] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5792.360899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5792.401798] alg: No test for adler32 (adler32-zlib) [ 5793.319750] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 5793.504633] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 5795.159234] Key type lgssc registered [ 5795.945369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5801.171854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5801.182838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.392817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5802.410963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5802.524213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.582159] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5812.388160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.439656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5812.451957] Lustre: 254596:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5812.636822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.802399] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5821.000420] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5824.806549] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5829.623686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.761382] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5829.904746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.831247] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5835.811767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5838.901794] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5886.954232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.961753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5886.971500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5886.980180] Lustre: Skipped 1 previous similar message [ 5889.875818] Lustre: server umount lustre-OST0000 complete [ 5893.053545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5901.694267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.921086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.382186] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5907.050717] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.063586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.063892] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5907.087612] Lustre: Skipped 1 previous similar message [ 5911.437314] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5948.387911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5948.392223] LustreError: Skipped 1 previous similar message [ 5948.395167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.403302] Lustre: Skipped 1 previous similar message [ 5948.411671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5953.512415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5953.518395] Lustre: Skipped 2 previous similar messages [ 5954.553393] Lustre: server umount lustre-OST0000 complete [ 5958.626153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.629476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5958.643702] Lustre: Skipped 2 previous similar messages [ 5958.657431] Lustre: Skipped 1 previous similar message [ 5963.775497] Lustre: server umount lustre-MDT0000 complete [ 5966.631399] LustreError: 254589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627825 with bad export cookie 3835519806910700189 [ 5966.631689] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5966.638694] LustreError: 254589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5966.888795] Lustre: server umount lustre-MDT0001 complete [ 5974.953845] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 5977.272283] Key type lgssc unregistered [ 5977.505467] LNet: 260010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5977.513787] LNetError: 260010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5977.528834] LNet: Removed LNI 192.168.202.156@tcp [ 5978.134645] Key type .llcrypt unregistered [ 5978.136736] Key type ._llcrypt unregistered [ 5995.871307] Key type ._llcrypt registered [ 5995.873643] Key type .llcrypt registered [ 5995.930505] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5996.628789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5996.669686] alg: No test for adler32 (adler32-zlib) [ 5997.669893] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 5997.842904] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 5999.473212] Key type lgssc registered [ 6000.489466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6007.116499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6007.140236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.573939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.760923] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6018.023619] LustreError: 261407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.041058] LustreError: 261407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6018.471316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.774784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.004105] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6028.586891] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6032.966924] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.299836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.593424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6043.453874] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6048.873855] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6052.908091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6069.218200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.221260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.231171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.313952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6073.315180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.319061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.333217] Lustre: Skipped 1 previous similar message [ 6075.324735] Lustre: server umount lustre-OST0000 complete [ 6078.432162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.433442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6078.440353] Lustre: Skipped 1 previous similar message [ 6078.453465] Lustre: Skipped 1 previous similar message [ 6083.552951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.558817] Lustre: Skipped 1 previous similar message [ 6088.673816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.677059] Lustre: Skipped 1 previous similar message [ 6092.255136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.389687] Lustre: server umount lustre-MDT0000 complete [ 6093.798376] LustreError: 261403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.811791] LustreError: 261403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6094.778045] LustreError: 261386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781627953 with bad export cookie 9981171623158121987 [ 6094.782729] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.784615] LustreError: 261386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6095.018300] Lustre: server umount lustre-MDT0001 complete [ 6102.624694] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6104.455613] Key type lgssc unregistered [ 6104.651315] LNet: 265712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.656887] LNetError: 265712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.671164] LNet: Removed LNI 192.168.202.156@tcp [ 6105.272380] Key type .llcrypt unregistered [ 6105.274514] Key type ._llcrypt unregistered [ 6115.811914] Key type ._llcrypt registered [ 6115.813641] Key type .llcrypt registered [ 6115.876798] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6116.227965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6116.272058] alg: No test for adler32 (adler32-zlib) [ 6117.247462] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6117.409768] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6119.041408] Key type lgssc registered [ 6119.788859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.415147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.426096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.709391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.380610] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6133.645249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.849625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.207193] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6140.465912] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6143.538907] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6147.713481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.906194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.025379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6151.443511] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6156.286741] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6168.546776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.554244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.564769] Lustre: Skipped 1 previous similar message [ 6173.645576] Lustre: server umount lustre-OST0000 complete [ 6178.786445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.790845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.802732] Lustre: Skipped 1 previous similar message [ 6178.814774] Lustre: Skipped 2 previous similar messages [ 6182.382845] Lustre: server umount lustre-MDT0000 complete [ 6183.907724] LustreError: 266911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.923923] LustreError: 266911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6184.739629] LustreError: 266892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628043 with bad export cookie 10530382281156658116 [ 6184.741684] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.746803] LustreError: 266892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6184.990436] Lustre: server umount lustre-MDT0001 complete [ 6191.488493] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6193.173423] Key type lgssc unregistered [ 6193.387083] LNet: 270585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6193.391146] LNetError: 270585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6193.401903] LNet: Removed LNI 192.168.202.156@tcp [ 6193.881186] Key type .llcrypt unregistered [ 6193.883217] Key type ._llcrypt unregistered [ 6209.233725] Key type ._llcrypt registered [ 6209.235315] Key type .llcrypt registered [ 6209.290608] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6210.031052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6210.051836] alg: No test for adler32 (adler32-zlib) [ 6211.022419] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6211.174729] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6212.807874] Key type lgssc registered [ 6213.515098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6218.331961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6218.349806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.672703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.725619] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6228.965490] LustreError: 271980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.981566] LustreError: 271980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6229.200734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.484728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6232.464056] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6237.494660] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6241.295534] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6245.564296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.755323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6249.735774] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6254.581402] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6264.185575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6279.651997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.655831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.656828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.663076] Lustre: Skipped 1 previous similar message [ 6279.672315] LustreError: Skipped 1 previous similar message [ 6284.156399] Lustre: server umount lustre-OST0000 complete [ 6289.888830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.891930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.895928] Lustre: Skipped 1 previous similar message [ 6289.900500] Lustre: Skipped 2 previous similar messages [ 6292.383956] Lustre: server umount lustre-MDT0000 complete [ 6294.338930] LustreError: 271961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628153 with bad export cookie 3821762955873180753 [ 6294.340061] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.346452] LustreError: 271961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6294.554831] Lustre: server umount lustre-MDT0001 complete [ 6300.864553] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6302.419949] Key type lgssc unregistered [ 6302.583742] LNet: 276300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6302.586628] LNetError: 276300:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6302.596613] LNet: Removed LNI 192.168.202.156@tcp [ 6303.024598] Key type .llcrypt unregistered [ 6303.026628] Key type ._llcrypt unregistered [ 6313.399926] Key type ._llcrypt registered [ 6313.402053] Key type .llcrypt registered [ 6313.491326] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6314.026334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.037805] alg: No test for adler32 (adler32-zlib) [ 6314.986655] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6315.103361] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6316.727278] Key type lgssc registered [ 6317.351855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6321.618246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6321.632256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.924156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.464596] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6330.171660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.396388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6332.761693] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6337.151285] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6340.678714] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6344.560442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6344.774894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.898019] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6348.921131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6352.126224] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6364.640966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.644345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.650145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6365.667081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6365.667495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6365.674399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.589732] Lustre: server umount lustre-OST0000 complete [ 6371.297707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6371.301094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.310588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6371.313085] Lustre: Skipped 1 previous similar message [ 6375.904452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6375.905056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.025948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6381.031236] Lustre: Skipped 2 previous similar messages [ 6385.119134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6385.217897] Lustre: server umount lustre-MDT0000 complete [ 6387.391925] LustreError: 278276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628246 with bad export cookie 6044928411357744302 [ 6387.395261] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.399170] LustreError: 278276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6387.606552] Lustre: server umount lustre-MDT0001 complete [ 6392.831699] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6394.492094] Key type lgssc unregistered [ 6394.717100] LNet: 281213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6394.721060] LNetError: 281213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6395.752590] LNet: Removed LNI 192.168.202.156@tcp [ 6396.255096] Key type .llcrypt unregistered [ 6396.256863] Key type ._llcrypt unregistered [ 6401.853175] Key type ._llcrypt registered [ 6401.854737] Key type .llcrypt registered [ 6401.899467] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6410.606549] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6419.049786] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6419.525196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6419.578903] alg: No test for adler32 (adler32-zlib) [ 6420.527048] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6420.674540] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6422.279175] Key type lgssc registered [ 6422.869382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6440.498887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6440.517571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.659981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6441.680860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6441.736374] Lustre: lustre-MDT0000: new disk, initializing [ 6441.785508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.797227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6443.732924] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6449.612092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.673617] Lustre: 284739:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6449.703557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6449.706726] Lustre: Skipped 1 previous similar message [ 6449.746067] Lustre: lustre-MDT0001: new disk, initializing [ 6449.786102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6449.804166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6449.808962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6451.631573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6456.646916] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6459.635647] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6463.492444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6463.674767] Lustre: lustre-OST0000: new disk, initializing [ 6463.679580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6463.684071] Lustre: 286708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6463.724436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6466.455768] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6467.527994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6467.534455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6467.575895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6471.541886] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.546131] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.644832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6475.723931] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6475.830879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6477.795775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.795804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.797071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6477.801507] Lustre: Skipped 1 previous similar message [ 6477.812566] LustreError: Skipped 1 previous similar message [ 6482.914138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.917014] Lustre: Skipped 2 previous similar messages [ 6488.033777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6488.035990] Lustre: Skipped 1 previous similar message [ 6491.103145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6491.188285] Lustre: server umount lustre-OST0000 complete [ 6493.157802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.168143] Lustre: Skipped 1 previous similar message [ 6493.174843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.180322] Lustre: Skipped 1 previous similar message [ 6498.272846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6498.279548] Lustre: Skipped 1 previous similar message [ 6499.338578] Lustre: server umount lustre-MDT0000 complete [ 6501.180864] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628360 with bad export cookie 3851263939441768983 [ 6501.184148] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.192926] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6501.338564] Lustre: server umount lustre-MDT0001 complete [ 6505.495860] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:46:04 (1781628364) [ 6509.298235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.555518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.504057] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6514.657925] LustreError: 289119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.669805] LustreError: 289119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6515.922109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.129341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.343491] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6522.264392] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6525.247168] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6528.588404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6528.783258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6531.778147] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6535.859417] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6546.403235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6546.405637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.415558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6546.423268] Lustre: Skipped 1 previous similar message [ 6546.431356] Lustre: Skipped 1 previous similar message [ 6547.181571] Lustre: server umount lustre-OST0000 complete [ 6551.521204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6551.536328] Lustre: Skipped 1 previous similar message [ 6555.451346] Lustre: server umount lustre-MDT0000 complete [ 6556.642583] LustreError: 290550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.649577] LustreError: 290550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6557.455759] LustreError: 289098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628416 with bad export cookie 3851263939441769872 [ 6557.458873] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6557.461285] LustreError: 289098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6557.604127] Lustre: server umount lustre-MDT0001 complete [ 6562.616851] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6564.089030] Key type lgssc unregistered [ 6564.232938] LNet: 292648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6564.236454] LNetError: 292648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6564.245927] LNet: Removed LNI 192.168.202.156@tcp [ 6564.628787] Key type .llcrypt unregistered [ 6564.630142] Key type ._llcrypt unregistered [ 6577.824385] Key type ._llcrypt registered [ 6577.825804] Key type .llcrypt registered [ 6577.874455] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6585.814613] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6586.260235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6586.318348] alg: No test for adler32 (adler32-zlib) [ 6587.219653] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6587.328412] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6588.959134] Key type lgssc registered [ 6589.461715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.948805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.958147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.075295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6611.094541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6611.141586] Lustre: lustre-MDT0000: new disk, initializing [ 6611.186754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6611.202303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6613.350863] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6619.129286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.214758] Lustre: 296463:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6619.248802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6619.251233] Lustre: Skipped 1 previous similar message [ 6619.295208] Lustre: lustre-MDT0001: new disk, initializing [ 6619.354086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6619.376734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6619.380877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6621.068394] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6626.011174] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6628.807233] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6632.223887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.371991] Lustre: lustre-OST2710: new disk, initializing [ 6632.373846] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6632.378608] Lustre: 298429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6632.432787] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6635.299193] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6639.634247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6639.640691] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6639.692936] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6640.292178] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6643.658171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.725273] Lustre: lustre-OST03e8: new disk, initializing [ 6643.727849] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6643.731495] Lustre: 299493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6643.784608] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6646.242719] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6649.282879] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6649.366150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6649.376238] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6649.432636] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6653.274797] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6653.370048] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6655.268349] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6655.389197] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6657.282225] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6657.381487] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6659.680810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6659.778674] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.674228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.680637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6664.688606] Lustre: Skipped 3 previous similar messages [ 6664.693593] Lustre: Skipped 2 previous similar messages [ 6667.912304] Lustre: server umount lustre-MDT0000 complete [ 6669.982897] LustreError: 297256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628529 with bad export cookie 4537011167069324508 [ 6669.983558] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.991273] LustreError: 297256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6670.192237] Lustre: server umount lustre-MDT0001 complete [ 6682.128208] Lustre: server umount lustre-OST2710 complete [ 6693.119257] Lustre: server umount lustre-OST03e8 complete [ 6700.973120] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6705.107514] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6721.100822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.202583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6721.222844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6721.265065] Lustre: lustre-MDT0000: new disk, initializing [ 6721.307658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.319232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6723.139626] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6728.308078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.364426] Lustre: 304736:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6728.487958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.350829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6734.980845] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6737.441952] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6740.604822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.722669] Lustre: lustre-OST0000: new disk, initializing [ 6740.724525] Lustre: Skipped 1 previous similar message [ 6740.726284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6740.729025] Lustre: Skipped 2 previous similar messages [ 6740.731812] Lustre: 306703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.769333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.772258] Lustre: Skipped 1 previous similar message [ 6743.605668] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6747.643983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6747.647189] Lustre: Skipped 1 previous similar message [ 6747.649524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6747.682325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6748.224493] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6750.372486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6750.510976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6752.484809] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6752.590387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6757.858383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.859225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6757.864362] LustreError: Skipped 1 previous similar message [ 6757.864953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.869131] Lustre: Skipped 1 previous similar message [ 6759.587050] Lustre: server umount lustre-OST0000 complete [ 6762.978581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.979496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6762.984590] Lustre: Skipped 2 previous similar messages [ 6767.397636] Lustre: server umount lustre-MDT0000 complete [ 6768.096975] LustreError: 304748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6768.105767] LustreError: 304748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6769.001285] LustreError: 304729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628628 with bad export cookie 4537011167069326174 [ 6769.004413] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6769.006118] LustreError: 304729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6769.139562] Lustre: server umount lustre-MDT0001 complete [ 6772.931849] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:50:31 (1781628631) [ 6773.738366] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6774.519951] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:50:33 (1781628633) [ 6775.836396] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6787.302625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.458276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6787.469480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6787.557276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6789.426253] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6793.125338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.182628] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6793.195690] Lustre: 310553:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6794.960745] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6811.619389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.701449] Lustre: Denying initial registration attempt from nid 192.168.202.156@tcp, specified as failover [ 6811.704479] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6811.708757] LustreError: 312006:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 6811.712447] LustreError: 312006:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6811.715091] LustreError: 312006:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6811.747176] Lustre: server umount lustre-OST0000 complete [ 6811.748892] LustreError: 312006:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6818.785076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.785080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6818.786144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.795701] Lustre: Skipped 2 previous similar messages [ 6820.566150] Lustre: server umount lustre-MDT0000 complete [ 6821.902321] LustreError: 310544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628681 with bad export cookie 4537011167069327042 [ 6821.905412] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6821.906306] LustreError: 310544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6822.018203] Lustre: server umount lustre-MDT0001 complete [ 6825.863258] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6827.010984] Key type lgssc unregistered [ 6827.156331] LNet: 313126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6827.159159] LNetError: 313126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6827.171318] LNet: Removed LNI 192.168.202.156@tcp [ 6827.492177] Key type .llcrypt unregistered [ 6827.494136] Key type ._llcrypt unregistered [ 6831.617557] Key type ._llcrypt registered [ 6831.618658] Key type .llcrypt registered [ 6831.658891] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6832.031321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.087929] alg: No test for adler32 (adler32-zlib) [ 6832.943975] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6833.038746] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6834.623193] Key type lgssc registered [ 6835.049405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6860.258989] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6863.702981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6863.710777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.848810] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6864.858406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6864.940050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.578563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6869.876507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.930313] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6869.943594] Lustre: 315864:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6870.063615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6871.602510] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6874.323688] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6876.448149] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6892.258991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6892.336518] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6892.412951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.607518] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6897.344945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6898.184431] Lustre: server umount lustre-OST0000 complete [ 6899.696256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.697235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.701289] Lustre: Skipped 1 previous similar message [ 6904.800070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.801929] Lustre: Skipped 2 previous similar messages [ 6905.757734] Lustre: server umount lustre-MDT0000 complete [ 6907.142508] LustreError: 315858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628766 with bad export cookie 16163011272884470001 [ 6907.147630] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6907.148411] LustreError: 315858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6907.276549] Lustre: server umount lustre-MDT0001 complete [ 6911.123209] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6912.186423] Key type lgssc unregistered [ 6912.334431] LNet: 319427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6912.337930] LNetError: 319427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6912.343319] LNet: Removed LNI 192.168.202.156@tcp [ 6912.644182] Key type .llcrypt unregistered [ 6912.645382] Key type ._llcrypt unregistered [ 6922.514957] Key type ._llcrypt registered [ 6922.516492] Key type .llcrypt registered [ 6922.556321] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6922.978483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.987391] alg: No test for adler32 (adler32-zlib) [ 6923.847839] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 6923.943545] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6925.527170] Key type lgssc registered [ 6925.914639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6928.840568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6928.847481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.020454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.454513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6934.563201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.701708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6936.157418] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6938.748704] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6940.636471] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6944.994565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.109056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.093953] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6949.718570] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6963.500858] LustreError: 323421:0:(ldlm_resource.c:1180:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (0000000060345fa2) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6964.708456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.709479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.714036] Lustre: Skipped 1 previous similar message [ 6966.416726] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 6966.419019] Lustre: Skipped 2 previous similar messages [ 6969.622293] Lustre: server umount lustre-MDT0000 complete [ 6969.825396] LustreError: 322251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.832227] LustreError: 322251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6970.907134] LustreError: 320801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628830 with bad export cookie 6086921687091604004 [ 6970.911371] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6970.911920] LustreError: 320801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6971.530294] LustreError: 322252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.537498] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 6974.943890] LustreError: 320817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.944099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.952650] Lustre: Skipped 1 previous similar message [ 6974.954685] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6981.088033] LustreError: 320815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.088396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6981.093751] LustreError: 320815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6981.097024] Lustre: Skipped 2 previous similar messages [ 6985.183124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6985.277493] Lustre: server umount lustre-MDT0001 complete [ 6987.750491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6990.693280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.447616] LustreError: 320131:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b9b0aef2 x1868173182789504/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6996.556637] LustreError: 322651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.562151] LustreError: 322651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6996.579400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.601643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6997.987069] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7000.971136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.112026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7002.602909] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7005.218576] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7011.301579] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7014.009354] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 7015.877172] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7016.441124] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7021.540842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.540941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.546767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.550694] Lustre: Skipped 1 previous similar message [ 7021.552997] Lustre: Skipped 1 previous similar message [ 7032.287102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7032.347446] Lustre: server umount lustre-OST0000 complete [ 7036.896823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.896864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7036.902181] Lustre: Skipped 1 previous similar message [ 7042.017104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7042.020469] Lustre: Skipped 7 previous similar messages [ 7048.159122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7048.216491] Lustre: server umount lustre-MDT0000 complete [ 7049.518257] LustreError: 320802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628908 with bad export cookie 6086921687091611123 [ 7049.521311] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7049.521740] LustreError: 320802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7049.652918] Lustre: server umount lustre-MDT0001 complete [ 7053.205733] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7054.280500] Key type lgssc unregistered [ 7054.414396] LNet: 327108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7054.417290] LNetError: 327108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7054.428328] LNet: Removed LNI 192.168.202.156@tcp [ 7054.728530] Key type .llcrypt unregistered [ 7054.729821] Key type ._llcrypt unregistered [ 7063.944661] Key type ._llcrypt registered [ 7063.945684] Key type .llcrypt registered [ 7063.982526] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7064.273762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.284367] alg: No test for adler32 (adler32-zlib) [ 7065.144800] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7065.228403] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7066.815130] Key type lgssc registered [ 7067.178246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7070.023074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7070.029348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.203857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.715033] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7075.796604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.947655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.383391] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7079.904979] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7081.832162] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7086.047863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7086.050838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.055682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7086.560239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.565750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.584278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.586115] Lustre: Skipped 1 previous similar message [ 7092.704098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7092.706271] Lustre: Skipped 1 previous similar message [ 7097.311118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7097.375949] Lustre: server umount lustre-MDT0000 complete [ 7097.824060] LustreError: 328502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.830117] LustreError: 328502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7098.606259] LustreError: 328482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781628957 with bad export cookie 5307547145282196007 [ 7098.609258] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.610307] LustreError: 328482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7098.718637] Lustre: server umount lustre-MDT0001 complete [ 7103.549598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.644205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7103.656997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7103.732854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.048847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7107.976774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.009865] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7108.021422] Lustre: 331085:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.116113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7109.553240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7112.160719] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7114.162771] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7119.011567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.103529] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7119.108926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7119.112682] LustreError: 332966:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 7119.115832] LustreError: 332966:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7119.118406] LustreError: 332966:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7119.151580] Lustre: server umount lustre-OST0000 complete [ 7119.153784] LustreError: 332966:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7124.322572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.409854] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7124.481693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.525584] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7129.106877] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7131.990861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.043157] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7132.077497] Lustre: lustre-OST0001: new disk, initializing [ 7132.079522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7132.082578] Lustre: 334496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7133.147356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7133.152103] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7133.197411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7133.226409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7134.358631] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7136.949278] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7138.273145] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.276242] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.277530] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7138.281833] Lustre: Skipped 1 previous similar message [ 7140.099858] Lustre: server umount lustre-OST0001 complete [ 7143.393878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7143.394558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.394873] LustreError: 334649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.396582] LustreError: Skipped 1 previous similar message [ 7143.406698] Lustre: Skipped 1 previous similar message [ 7147.478095] Lustre: server umount lustre-OST0000 complete [ 7149.023816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7149.026513] LustreError: Skipped 1 previous similar message [ 7149.028037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.032602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.034731] Lustre: Skipped 3 previous similar messages [ 7163.359129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7163.420823] Lustre: server umount lustre-MDT0000 complete [ 7163.872329] LustreError: 333453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.877312] LustreError: 333453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7164.722021] LustreError: 331077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629023 with bad export cookie 5307547145282196546 [ 7164.725186] LustreError: 331077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7164.725659] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7164.849189] Lustre: server umount lustre-MDT0001 complete [ 7168.695810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7169.753136] Key type lgssc unregistered [ 7169.896481] LNet: 336457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7169.900944] LNetError: 336457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7169.909271] LNet: Removed LNI 192.168.202.156@tcp [ 7170.239310] Key type .llcrypt unregistered [ 7170.240533] Key type ._llcrypt unregistered [ 7201.524835] Key type ._llcrypt registered [ 7201.526360] Key type .llcrypt registered [ 7201.570811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7208.275363] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7208.739661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7208.747475] alg: No test for adler32 (adler32-zlib) [ 7209.608855] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7209.692323] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7211.271156] Key type lgssc registered [ 7211.637805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7224.021537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7224.026612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.136462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7225.152664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7225.194650] Lustre: lustre-MDT0000: new disk, initializing [ 7225.232249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.241762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7226.736964] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7230.892708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.938877] Lustre: 341359:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7230.963328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7230.965327] Lustre: Skipped 1 previous similar message [ 7231.005576] Lustre: lustre-MDT0001: new disk, initializing [ 7231.049116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.064420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7231.068752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7232.495790] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7236.229489] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7238.061536] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7240.434292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.546366] Lustre: lustre-OST0000: new disk, initializing [ 7240.547785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7240.550113] Lustre: 343325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7240.579029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.491293] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7246.281072] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7246.846843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7246.850896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7246.888769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7247.796773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7247.869063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7249.287660] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7249.369432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7251.937598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7251.938584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.939833] LustreError: Skipped 1 previous similar message [ 7251.940294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7251.944696] Lustre: Skipped 1 previous similar message [ 7256.150353] Lustre: server umount lustre-OST0000 complete [ 7261.663993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7261.665880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.669455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7261.671477] Lustre: Skipped 1 previous similar message [ 7263.571737] Lustre: server umount lustre-MDT0000 complete [ 7264.725105] LustreError: 341351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629123 with bad export cookie 13170829305215360739 [ 7264.727561] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7264.729355] LustreError: 341351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7264.851185] Lustre: server umount lustre-MDT0001 complete [ 7267.530294] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:58:46 (1781629126) [ 7270.119856] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:58:49 (1781629129) [ 7272.992970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.198787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.613276] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7277.562040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.684336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7279.053102] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7281.480692] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7283.314873] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7285.588469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.697237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.650573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7290.180609] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7305.359186] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7306.025156] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7306.594449] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7307.193514] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7307.895126] Lustre: Failing over lustre-MDT0000 [ 7308.031812] Lustre: server umount lustre-MDT0000 complete [ 7308.768905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7308.771555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.776502] Lustre: Skipped 1 previous similar message [ 7308.779040] LustreError: 346076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.785331] LustreError: 346076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7309.280139] LustreError: 346803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.286658] LustreError: 346803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7312.018730] LustreError: 346071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7314.400472] LustreError: 348566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7314.405971] LustreError: 348566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7319.520538] LustreError: 346803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.525679] LustreError: 346803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7321.476630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.524893] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7321.625233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.653545] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7322.252969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7323.302675] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7326.694965] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7326.706432] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7326.725793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7328.718986] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7329.309775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7331.254567] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7331.944046] Lustre: Failing over lustre-MDT0000 [ 7332.121978] Lustre: server umount lustre-MDT0000 complete [ 7332.502541] LustreError: 346071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.929480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.932560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7336.934020] Lustre: Skipped 4 previous similar messages [ 7345.332973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.371238] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7345.501529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7346.966464] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7347.850424] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7350.756715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7350.759694] Lustre: Skipped 2 previous similar messages [ 7350.769721] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7350.787838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7352.612749] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7353.220440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7355.872939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7355.874231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.880890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7355.883067] Lustre: Skipped 4 previous similar messages [ 7360.993267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7360.996410] Lustre: Skipped 2 previous similar messages [ 7361.843316] Lustre: server umount lustre-MDT0000 complete [ 7363.311609] LustreError: 346057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629222 with bad export cookie 13170829305215363420 [ 7363.312688] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.316634] LustreError: 346057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7363.442484] Lustre: server umount lustre-MDT0001 complete [ 7375.133952] Lustre: server umount lustre-OST0000 complete [ 7380.612942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.783235] LustreError: 352428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.789647] LustreError: 352428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7380.806148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.810355] Lustre: Skipped 1 previous similar message [ 7382.290236] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7385.358487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.044416] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7389.791501] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7391.830229] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7394.369224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.618937] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7399.288370] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7405.799805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7416.293750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.295060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7416.295323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7416.299578] Lustre: Skipped 3 previous similar messages [ 7417.311308] Lustre: server umount lustre-OST0000 complete [ 7424.799768] Lustre: server umount lustre-MDT0000 complete [ 7426.134165] LustreError: 352410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629285 with bad export cookie 13170829305215364141 [ 7426.135269] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7426.138529] LustreError: 352410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7426.258818] Lustre: server umount lustre-MDT0001 complete [ 7430.080933] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7431.104349] Key type lgssc unregistered [ 7431.233646] LNet: 355991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7431.236317] LNetError: 355991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.243321] LNet: Removed LNI 192.168.202.156@tcp [ 7431.572805] Key type .llcrypt unregistered [ 7431.574309] Key type ._llcrypt unregistered [ 7441.687783] Key type ._llcrypt registered [ 7441.689146] Key type .llcrypt registered [ 7441.741574] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7442.276094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.301102] alg: No test for adler32 (adler32-zlib) [ 7443.176418] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7443.270741] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7444.855138] Key type lgssc registered [ 7445.205434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7447.950449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7447.958246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.104939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.453432] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7453.495822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.613301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.954105] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7457.328384] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7459.123289] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7461.309894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7461.419444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7463.473477] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7465.919403] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7474.726644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7484.384304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7484.387393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7484.392959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.897343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7484.898400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.901430] Lustre: Skipped 1 previous similar message [ 7484.905286] Lustre: Skipped 1 previous similar message [ 7486.494265] Lustre: server umount lustre-MDT0000 complete [ 7487.815178] LustreError: 357464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629346 with bad export cookie 14191893970278082719 [ 7487.817513] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.820375] LustreError: 357464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7487.936987] Lustre: server umount lustre-MDT0001 complete [ 7499.550816] Lustre: server umount lustre-OST0000 complete [ 7504.210520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.364358] LustreError: 361210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.370516] LustreError: 361210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7504.385131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.741905] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7508.728729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.854997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7510.206199] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7512.622932] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7514.450602] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7516.864262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.046047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.214409] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7521.747690] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7529.317062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7539.681579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7539.682139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.683119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.690377] Lustre: Skipped 1 previous similar message [ 7544.801995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.804575] Lustre: Skipped 4 previous similar messages [ 7545.111443] Lustre: server umount lustre-MDT0000 complete [ 7546.428436] LustreError: 361945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629405 with bad export cookie 14191893970278083818 [ 7546.433161] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7546.433442] LustreError: 361945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7546.535420] Lustre: server umount lustre-MDT0001 complete [ 7558.238467] Lustre: server umount lustre-OST0000 complete [ 7563.806061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.947879] LustreError: 365139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.969913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.275023] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7568.158145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.640025] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7571.974456] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7573.876809] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7576.205445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.303227] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7580.679502] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7589.670233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7593.952925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7593.955226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.958844] Lustre: Skipped 1 previous similar message [ 7593.960736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7598.398638] Lustre: server umount lustre-OST0000 complete [ 7600.096866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.102247] Lustre: Skipped 2 previous similar messages [ 7605.216235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.218661] Lustre: Skipped 5 previous similar messages [ 7605.914138] Lustre: server umount lustre-MDT0000 complete [ 7607.244714] LustreError: 365121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629466 with bad export cookie 14191893970278084826 [ 7607.245933] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7607.248535] LustreError: 365121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7607.373414] Lustre: server umount lustre-MDT0001 complete [ 7611.236057] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7612.387910] Key type lgssc unregistered [ 7612.509061] LNet: 368689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7612.511897] LNetError: 368689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7612.522391] LNet: Removed LNI 192.168.202.156@tcp [ 7612.870253] Key type .llcrypt unregistered [ 7612.871428] Key type ._llcrypt unregistered [ 7622.897241] Key type ._llcrypt registered [ 7622.898791] Key type .llcrypt registered [ 7622.935213] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7623.237038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7623.244226] alg: No test for adler32 (adler32-zlib) [ 7624.105860] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7624.191565] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7625.783166] Key type lgssc registered [ 7626.160568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.884157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7628.891362] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.894518] LustreError: 370105:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7628.903157] LustreError: 370105:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7628.906176] LustreError: 370105:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7628.908767] LustreError: 370105:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7628.911573] LustreError: 370105:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7631.583152] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7631.586875] LustreError: 370502:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7631.599094] LustreError: 370502:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7631.601936] LustreError: 370502:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7631.604672] LustreError: 370502:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7631.607750] LustreError: 370502:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7638.543339] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7639.695838] Key type lgssc unregistered [ 7639.849312] LNet: 371504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7639.851573] LNetError: 371504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7639.865584] LNet: Removed LNI 192.168.202.156@tcp [ 7640.187476] Key type .llcrypt unregistered [ 7640.189043] Key type ._llcrypt unregistered [ 7646.961825] Key type ._llcrypt registered [ 7646.963902] Key type .llcrypt registered [ 7647.013597] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7653.511895] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7653.941089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7653.947790] alg: No test for adler32 (adler32-zlib) [ 7654.850214] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7654.945629] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7656.543164] Key type lgssc registered [ 7656.990829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7669.627717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7669.634874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.729680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7670.742417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7670.779629] Lustre: lustre-MDT0000: new disk, initializing [ 7670.812285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.820800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7672.249706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7676.531028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.574563] Lustre: 374633:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7676.599399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7676.601425] Lustre: Skipped 1 previous similar message [ 7676.639209] Lustre: lustre-MDT0001: new disk, initializing [ 7676.674210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7676.688097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7676.691785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7678.150380] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7682.140489] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7684.129564] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7686.741625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7686.876643] Lustre: lustre-OST0000: new disk, initializing [ 7686.878768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7686.881757] Lustre: 376602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7686.919713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.443269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7688.447241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7688.491762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7689.063364] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7692.962911] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7694.615850] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7694.700931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7696.392900] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7696.475870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7698.914501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7698.914603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.915515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7698.918141] LustreError: Skipped 1 previous similar message [ 7698.922557] Lustre: Skipped 1 previous similar message [ 7703.316663] Lustre: server umount lustre-OST0000 complete [ 7707.616099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7707.619290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.624561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.626620] Lustre: Skipped 1 previous similar message [ 7709.152993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7709.153400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.272730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.274451] Lustre: Skipped 2 previous similar messages [ 7718.879143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7719.034285] Lustre: server umount lustre-MDT0000 complete [ 7721.868034] LustreError: 374625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629580 with bad export cookie 13493499169412171375 [ 7721.871746] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7721.880210] LustreError: 374625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7722.139321] Lustre: server umount lustre-MDT0001 complete [ 7728.836979] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7729.935183] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:06:28 (1781629588) [ 7735.400375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.674150] LustreError: 379059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7735.685421] LustreError: 379059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7735.733250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.978308] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7740.897889] LustreError: 379060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.998040] LustreError: 379059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.554298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.799848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7749.560883] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7755.805264] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7759.488771] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7764.275642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7764.448045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.099460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7773.807639] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7777.557453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.670904] Lustre: lustre-OST0001: new disk, initializing [ 7777.673481] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7777.679671] Lustre: 381895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7780.791899] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7784.471018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7784.475314] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7784.523115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7786.307320] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7789.540454] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.545411] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.552235] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7789.554795] Lustre: Skipped 1 previous similar message [ 7790.856171] Lustre: server umount lustre-OST0001 complete [ 7791.901368] LustreError: 380879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.779436] LustreError: 380876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.785185] LustreError: 380876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7804.901022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.902652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.906544] Lustre: Skipped 2 previous similar messages [ 7804.911204] Lustre: Skipped 1 previous similar message [ 7810.021779] LustreError: 380878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.030462] LustreError: 380878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7818.207125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7818.295890] Lustre: server umount lustre-OST0000 complete [ 7822.816569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7822.821114] LustreError: Skipped 1 previous similar message [ 7822.830704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.843652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.848422] Lustre: Skipped 4 previous similar messages [ 7826.623734] Lustre: server umount lustre-MDT0000 complete [ 7828.786127] LustreError: 379040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629687 with bad export cookie 13493499169412172236 [ 7828.788312] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.793811] LustreError: 379040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7843.295285] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7843.416397] Lustre: server umount lustre-MDT0001 complete [ 7849.167481] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7850.569862] Key type lgssc unregistered [ 7850.752653] LNet: 383879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7850.756686] LNetError: 383879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7850.768492] LNet: Removed LNI 192.168.202.156@tcp [ 7851.177805] Key type .llcrypt unregistered [ 7851.179464] Key type ._llcrypt unregistered [ 7876.892760] Key type ._llcrypt registered [ 7876.894242] Key type .llcrypt registered [ 7876.964137] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7877.410961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.457740] alg: No test for adler32 (adler32-zlib) [ 7878.459780] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 7878.616923] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7880.231203] Key type lgssc registered [ 7880.706614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7884.993239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7885.009316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.148178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7886.161722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7886.258482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.050864] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7894.305112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.356639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7894.373242] Lustre: 386654:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7894.584790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7897.132646] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7901.761159] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7904.751616] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7909.857038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7909.862043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.883198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7910.368876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.387066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7910.392775] Lustre: Skipped 1 previous similar message [ 7912.343848] Lustre: server umount lustre-MDT0000 complete [ 7914.762020] LustreError: 386646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629773 with bad export cookie 2060253378990037547 [ 7914.772173] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7914.774615] LustreError: 386646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7914.974534] Lustre: server umount lustre-MDT0001 complete [ 7921.199399] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:09:39 (1781629779) [ 7925.719219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.012952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.667981] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7931.360275] LustreError: 389164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.369682] LustreError: 389164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7933.860810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.043713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7936.563387] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7940.858769] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7943.905613] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7947.857723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.973389] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7948.066843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.258705] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7955.156292] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7960.537428] Lustre: Permanently deactivating lustre-OST0000 [ 7960.542898] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7961.211890] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7962.673866] LustreError: 392001:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7962.679703] LustreError: 392001:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7962.682456] LustreError: 392001:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7963.494608] LustreError: 392049:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7963.497972] LustreError: 392049:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7970.388986] Lustre: server umount lustre-OST0000 complete [ 7971.635045] LustreError: 392299:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7971.637508] LustreError: 392299:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7974.880778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.881431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.886186] Lustre: Skipped 1 previous similar message [ 7980.000100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.002703] Lustre: Skipped 2 previous similar messages [ 7982.072904] Lustre: server umount lustre-MDT0000 complete [ 7983.482194] LustreError: 391851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629842 with bad export cookie 2060253378990038114 [ 7983.485804] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.486671] LustreError: 391851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7993.875644] Lustre: server umount lustre-MDT0001 complete [ 7996.869715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.280507] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7999.174920] LustreError: 393456:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8001.276367] Lustre: 393796:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.156@tcp [ 8001.938375] Lustre: server umount MGS complete [ 8005.658200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.804409] LustreError: 394519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.811607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8005.814401] Lustre: Skipped 1 previous similar message [ 8005.833266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8007.146189] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8010.006152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.113382] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8011.451651] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8013.931771] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8015.738588] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8016.421505] Lustre: Permanently reactivating lustre-OST0000 [ 8016.427228] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8016.429890] Lustre: Skipped 2 previous similar messages [ 8039.903823] LustreError: 396251:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8042.686474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.828962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.831205] Lustre: Skipped 1 previous similar message [ 8044.910866] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8047.432923] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8059.360114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8059.362914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.367594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8060.386087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8061.330426] Lustre: server umount lustre-OST0000 complete [ 8064.723781] Lustre: server umount lustre-MDT0000 complete [ 8065.505887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.509505] Lustre: Skipped 1 previous similar message [ 8065.511230] LustreError: 394519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.516430] LustreError: 394519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8066.062663] LustreError: 394500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629925 with bad export cookie 2060253378990039136 [ 8066.065665] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8066.066671] LustreError: 394500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8066.191950] Lustre: server umount lustre-MDT0001 complete [ 8069.884810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8070.931790] Key type lgssc unregistered [ 8071.071420] LNet: 399092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8071.073627] LNetError: 399092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8071.081286] LNet: Removed LNI 192.168.202.156@tcp [ 8071.414973] Key type .llcrypt unregistered [ 8071.416064] Key type ._llcrypt unregistered [ 8077.187366] Key type ._llcrypt registered [ 8077.188345] Key type .llcrypt registered [ 8077.227072] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8083.547830] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8083.953597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8083.998161] alg: No test for adler32 (adler32-zlib) [ 8084.852314] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8084.949603] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8086.535164] Key type lgssc registered [ 8086.913862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8097.740358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8097.746705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8098.835405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8098.844917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8098.884869] Lustre: lustre-MDT0000: new disk, initializing [ 8098.909698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.917077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8100.197719] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8104.376618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.421331] Lustre: 402222:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8104.447340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8104.451029] Lustre: Skipped 1 previous similar message [ 8104.489386] Lustre: lustre-MDT0001: new disk, initializing [ 8104.520431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8104.532134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8104.535634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8105.850229] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8109.614579] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8111.337753] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8113.639942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8113.740853] Lustre: lustre-OST0000: new disk, initializing [ 8113.742658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8113.745144] Lustre: 404188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.771327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.616890] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8119.202171] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8119.796193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8119.798973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8119.839438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8120.679206] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8120.753763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8122.189743] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8122.266775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8124.897297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8124.898746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.899474] LustreError: Skipped 1 previous similar message [ 8124.903112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8124.906188] Lustre: Skipped 1 previous similar message [ 8129.044198] Lustre: server umount lustre-OST0000 complete [ 8135.137048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.137095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.137769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8135.145652] Lustre: Skipped 2 previous similar messages [ 8136.479606] Lustre: server umount lustre-MDT0000 complete [ 8137.762144] LustreError: 405725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781629996 with bad export cookie 5517345193433279635 [ 8137.765305] LustreError: 405725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8137.765639] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.884121] Lustre: server umount lustre-MDT0001 complete [ 8140.711840] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8141.272101] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:13:20 (1781630000) [ 8143.492294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.664231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.046688] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8147.912372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.032822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8149.385882] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8151.754633] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8153.541415] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8155.719735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.693207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8160.021732] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8160.656206] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8171.732121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8174.561717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.562961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.565896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8174.567062] LustreError: Skipped 1 previous similar message [ 8174.572135] Lustre: Skipped 2 previous similar messages [ 8179.283052] Lustre: server umount lustre-OST0000 complete [ 8183.775837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.778461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.782316] Lustre: Skipped 1 previous similar message [ 8183.784065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8186.710334] Lustre: server umount lustre-MDT0000 complete [ 8187.914558] LustreError: 406625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630047 with bad export cookie 5517345193433280503 [ 8187.918235] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.919540] LustreError: 406625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8188.043396] Lustre: server umount lustre-MDT0001 complete [ 8191.532651] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8192.451725] Key type lgssc unregistered [ 8192.568387] LNet: 410383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8192.570679] LNetError: 410383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8192.576307] LNet: Removed LNI 192.168.202.156@tcp [ 8192.877315] Key type .llcrypt unregistered [ 8192.878530] Key type ._llcrypt unregistered [ 8200.592111] Key type ._llcrypt registered [ 8200.593559] Key type .llcrypt registered [ 8200.630634] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8200.727222] Key type .llcrypt unregistered [ 8200.728267] Key type ._llcrypt unregistered [ 8207.661219] Key type ._llcrypt registered [ 8207.662208] Key type .llcrypt registered [ 8207.696146] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8208.133311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8208.140935] alg: No test for adler32 (adler32-zlib) [ 8208.993183] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8209.089045] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8210.671127] Key type lgssc registered [ 8211.013879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8213.523780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8213.530868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.686390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.980206] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8218.763807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8218.901425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.831357] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8223.288674] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8224.033194] LustreError: 412570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8224.037632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8224.039755] LustreError: 412570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8225.606725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.734585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8227.168525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8234.463827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8234.466156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.470299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.001372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.001984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8236.005658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.342226] Lustre: server umount lustre-OST0000 complete [ 8244.705417] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8244.707885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.713219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.715768] Lustre: Skipped 1 previous similar message [ 8247.768851] Lustre: server umount lustre-MDT0000 complete [ 8248.950219] LustreError: 412550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630108 with bad export cookie 16414322004845311657 [ 8248.952982] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.955281] LustreError: 412550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8249.073281] Lustre: server umount lustre-MDT0001 complete [ 8252.729920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8253.830725] Key type lgssc unregistered [ 8253.973427] LNet: 415718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8253.976368] LNetError: 415718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8253.988404] LNet: Removed LNI 192.168.202.156@tcp [ 8254.306237] Key type .llcrypt unregistered [ 8254.308421] Key type ._llcrypt unregistered [ 8263.505183] Key type ._llcrypt registered [ 8263.506518] Key type .llcrypt registered [ 8263.548546] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8263.910709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8263.923828] alg: No test for adler32 (adler32-zlib) [ 8264.773133] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8264.874339] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8266.455113] Key type lgssc registered [ 8266.816478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8269.411609] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8269.417830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8284.831320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8289.951250] LustreError: 417085:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 8316.575194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8319.809207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8322.948370] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8325.312076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.484597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8325.509903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8326.860560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8329.729212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.857587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8331.155849] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8333.636225] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8335.489129] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8340.448499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.451647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.456089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.959899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.962518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.966568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.816762] Lustre: server umount lustre-OST0000 complete [ 8345.567887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8345.570227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.574293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.575743] Lustre: Skipped 2 previous similar messages [ 8351.064137] Lustre: server umount lustre-MDT0000 complete [ 8351.714240] LustreError: 418125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.718366] LustreError: 418125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8352.352097] LustreError: 417091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630211 with bad export cookie 8437252126049760157 [ 8352.355762] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8352.355927] LustreError: 417091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8352.469875] Lustre: server umount lustre-MDT0001 complete [ 8356.303156] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8357.355871] Key type lgssc unregistered [ 8357.502274] LNet: 420655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8357.505120] LNetError: 420655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8357.512318] LNet: Removed LNI 192.168.202.156@tcp [ 8357.799854] Key type .llcrypt unregistered [ 8357.801460] Key type ._llcrypt unregistered [ 8366.986404] Key type ._llcrypt registered [ 8366.987430] Key type .llcrypt registered [ 8367.024387] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8367.336589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8367.385724] alg: No test for adler32 (adler32-zlib) [ 8368.242927] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8368.322483] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8369.903104] Key type lgssc registered [ 8370.281989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8372.759101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8372.765985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.923702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.219688] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8378.041813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.171812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.541232] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8381.948120] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8383.753469] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8385.928876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.037572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.804038] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8390.119686] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8399.333504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8403.935909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8403.939954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.946973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8404.448669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8404.449372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.452075] Lustre: Skipped 1 previous similar message [ 8404.457852] Lustre: Skipped 1 previous similar message [ 8405.651674] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 8405.653919] Lustre: Skipped 1 previous similar message [ 8407.509579] Lustre: server umount lustre-MDT0000 complete [ 8408.346573] LustreError: 422029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.56@tcp arrived at 1781630267 with bad export cookie 5783719694646853403 [ 8408.351885] LustreError: 422029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8409.043985] LustreError: 422027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630268 with bad export cookie 5783719694646852752 [ 8409.047491] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8409.567874] LustreError: 422047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8409.568391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.572468] LustreError: 422047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8409.576057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.687932] LustreError: 424535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.688128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.692867] LustreError: 424535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8415.189555] Lustre: server umount lustre-OST0000 complete [ 8417.550708] Lustre: server umount lustre-MDT0001 complete [ 8421.148970] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8422.170563] Key type lgssc unregistered [ 8422.295382] LNet: 425748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8422.298382] LNetError: 425748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8422.306260] LNet: Removed LNI 192.168.202.156@tcp [ 8422.613097] Key type .llcrypt unregistered [ 8422.614686] Key type ._llcrypt unregistered [ 8431.674889] Key type ._llcrypt registered [ 8431.676476] Key type .llcrypt registered [ 8431.718286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8432.059720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8432.067640] alg: No test for adler32 (adler32-zlib) [ 8432.911715] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8432.993447] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8434.575160] Key type lgssc registered [ 8434.921070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8437.336076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8437.342944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8438.493872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.821708] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8442.515290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.626354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8443.993135] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8446.431870] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8448.258436] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8450.609473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.747908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8452.757078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8455.293829] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8464.103390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8468.448461] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8468.452749] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.456618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8469.473920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8469.476743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.186485] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 8471.189364] Lustre: Skipped 1 previous similar message [ 8474.593910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8474.595876] Lustre: Skipped 1 previous similar message [ 8479.713584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8479.715160] Lustre: Skipped 2 previous similar messages [ 8481.247152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8481.303246] Lustre: server umount lustre-MDT0001 complete [ 8481.418708] LustreError: 427165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.424154] LustreError: 427165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8484.832570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8484.832745] LustreError: 427141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.833608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.104344] Lustre: server umount lustre-OST0000 complete [ 8489.952255] LustreError: 427880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.434423] Lustre: server umount lustre-MDT0000 complete [ 8495.152178] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8496.239291] Key type lgssc unregistered [ 8496.370458] LNet: 430845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8496.373180] LNetError: 430845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8496.380312] LNet: Removed LNI 192.168.202.156@tcp [ 8496.678120] Key type .llcrypt unregistered [ 8496.679872] Key type ._llcrypt unregistered [ 8504.951217] Key type ._llcrypt registered [ 8504.952143] Key type .llcrypt registered [ 8504.992198] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8511.153219] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8511.509690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8511.566587] alg: No test for adler32 (adler32-zlib) [ 8512.430605] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8512.516489] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8514.103138] Key type lgssc registered [ 8514.479295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8526.395376] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8531.237307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8531.242386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.354210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8532.365687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8532.404575] Lustre: lustre-MDT0000: new disk, initializing [ 8532.427102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.434048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8533.780410] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8539.385380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.482783] Lustre: lustre-OST0000: new disk, initializing [ 8539.484465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8539.486082] Lustre: Skipped 1 previous similar message [ 8539.487875] Lustre: 435847:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8539.511267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.645678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8539.648736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8539.657691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8541.521209] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8545.291707] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8550.903633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.947331] Lustre: 434830:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8550.979114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8551.014553] Lustre: lustre-MDT0001: new disk, initializing [ 8551.060168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8551.075623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8551.079780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8552.433774] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8556.139059] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8556.220445] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8557.648721] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8557.721806] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8561.299740] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 8562.145686] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.146245] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8562.148981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8562.153058] Lustre: Skipped 1 previous similar message [ 8565.507692] Lustre: server umount lustre-MDT0001 complete [ 8566.413169] LustreError: 434836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.419131] LustreError: 434836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8567.266484] LustreError: 438417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.276811] LustreError: 438417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8572.386734] LustreError: 434838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.395669] LustreError: 434838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8577.509741] LustreError: 434837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.518820] LustreError: 434837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8580.576183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8580.580883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.588556] Lustre: Skipped 1 previous similar message [ 8580.591576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.624602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.626898] LustreError: 438417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.516231] Lustre: server umount lustre-OST0000 complete [ 8593.854585] Lustre: server umount lustre-MDT0000 complete [ 8598.851388] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8599.929838] Key type lgssc unregistered [ 8600.071452] LNet: 439594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8600.074070] LNetError: 439594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8600.080460] LNet: Removed LNI 192.168.202.156@tcp [ 8600.373786] Key type .llcrypt unregistered [ 8600.375051] Key type ._llcrypt unregistered [ 8615.930134] Key type ._llcrypt registered [ 8615.931845] Key type .llcrypt registered [ 8616.015880] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8616.164335] Key type .llcrypt unregistered [ 8616.165924] Key type ._llcrypt unregistered [ 8625.643879] Key type ._llcrypt registered [ 8625.645686] Key type .llcrypt registered [ 8625.733611] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8626.492806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8626.611317] alg: No test for adler32 (adler32-zlib) [ 8627.536259] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 8627.696818] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8629.319954] Key type lgssc registered [ 8629.976680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8641.737056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8641.750104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.860703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8642.873566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8642.911694] Lustre: lustre-MDT0000: new disk, initializing [ 8642.940540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.948160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8644.481378] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8649.424632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.474137] Lustre: 442943:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8649.509661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8649.512605] Lustre: Skipped 1 previous similar message [ 8649.554780] Lustre: lustre-MDT0001: new disk, initializing [ 8649.604901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8649.631958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8649.641372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8651.304772] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8655.582226] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8657.848808] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8660.373970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.513064] Lustre: lustre-OST0000: new disk, initializing [ 8660.514954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8660.517839] Lustre: 444910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8660.556046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.097430] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8667.073442] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8669.191507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8669.194863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8669.240558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8675.296037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8675.299108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.302990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.393245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.396043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.396817] Lustre: Skipped 1 previous similar message [ 8679.402074] Lustre: Skipped 1 previous similar message [ 8680.868129] Lustre: server umount lustre-MDT0000 complete [ 8682.434629] LustreError: 442936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630541 with bad export cookie 10279160783794359074 [ 8682.439086] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.439681] LustreError: 442936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8682.570255] Lustre: server umount lustre-MDT0001 complete [ 8694.189773] Lustre: server umount lustre-OST0000 complete [ 8698.309406] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:22:37 (1781630557) [ 8701.301335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8701.504437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.019247] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8706.173601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.309266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8707.757801] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8710.338319] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8712.162426] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8727.522016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8727.582159] Lustre: Found index 0 for lustre-OST0000, updating log [ 8727.637199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.542472] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8732.003583] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8747.109061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8752.612423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.622417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.623431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.631854] Lustre: Skipped 2 previous similar messages [ 8755.035504] Lustre: server umount lustre-OST0000 complete [ 8767.762261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.763097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8767.767823] Lustre: Skipped 2 previous similar messages [ 8767.769842] Lustre: Skipped 2 previous similar messages [ 8773.088026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8773.089718] Lustre: Skipped 2 previous similar messages [ 8782.303124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8782.357898] Lustre: server umount lustre-MDT0000 complete [ 8783.572192] LustreError: 448775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781630642 with bad export cookie 10279160783794360453 [ 8783.575296] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8783.575991] LustreError: 448775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8783.688209] Lustre: server umount lustre-MDT0001 complete [ 8786.383053] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:24:05 (1781630645) [ 8788.447268] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [ 9367.857893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.009222] LustreError: 453363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.016117] LustreError: 453363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9368.032726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.329904] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9372.038679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.148931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9373.412346] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9375.675131] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9377.355163] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9392.546542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9392.628341] Lustre: Found index 0 for lustre-OST0000, updating log [ 9392.716232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9394.693993] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9395.814488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9397.145725] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9398.882429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.887699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9398.890068] Lustre: Skipped 6 previous similar messages [ 9399.905993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9403.988669] Lustre: server umount lustre-OST0000 complete [ 9406.240600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.241797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9406.245551] Lustre: Skipped 1 previous similar message [ 9410.336327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.338688] Lustre: Skipped 8 previous similar messages [ 9411.346619] Lustre: server umount lustre-MDT0000 complete [ 9411.359951] LustreError: 454811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.365986] LustreError: 454811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9412.384084] LustreError: 453363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.391026] LustreError: 453363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9412.559419] LustreError: 453343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631271 with bad export cookie 10279160783794361517 [ 9412.562076] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9412.563513] LustreError: 453343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9412.675555] Lustre: server umount lustre-MDT0001 complete [ 9428.447300] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [ 9430.201567] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9433.359896] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9445.255778] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:35:04 (1781631304) [ 9451.116274] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9452.104826] Key type lgssc unregistered [ 9455.263499] LNet: 460013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9455.266328] LNetError: 460013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9524.901291] LNet: Removed LNI 192.168.202.156@tcp [ 9525.187478] Key type .llcrypt unregistered [ 9525.188663] Key type ._llcrypt unregistered [ 9531.557084] Key type ._llcrypt registered [ 9531.558133] Key type .llcrypt registered [ 9531.601458] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9531.971567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9532.005621] alg: No test for adler32 (adler32-zlib) [ 9532.875641] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9532.961298] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9534.543304] Key type lgssc registered [ 9534.964285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9536.459972] LNet: Added LNI 192.168.202.156@tcp100 [8/256/0/180] [ 9536.763698] LNet: Added LNI 192.168.202.156@tcp101 [8/256/0/180] [ 9537.056759] LNet: Added LNI 192.168.202.156@tcp102 [8/256/0/180] [ 9537.359183] LNet: Added LNI 192.168.202.156@tcp103 [8/256/0/180] [ 9537.651590] LNet: Added LNI 192.168.202.156@tcp104 [8/256/0/180] [ 9537.956363] LNet: Added LNI 192.168.202.156@tcp105 [8/256/0/180] [ 9538.233842] LNet: Added LNI 192.168.202.156@tcp106 [8/256/0/180] [ 9538.530960] LNet: Added LNI 192.168.202.156@tcp107 [8/256/0/180] [ 9538.836335] LNet: Added LNI 192.168.202.156@tcp108 [8/256/0/180] [ 9539.128573] LNet: Added LNI 192.168.202.156@tcp109 [8/256/0/180] [ 9539.438124] LNet: Added LNI 192.168.202.156@tcp110 [8/256/0/180] [ 9539.735338] LNet: Added LNI 192.168.202.156@tcp111 [8/256/0/180] [ 9540.032542] LNet: Added LNI 192.168.202.156@tcp112 [8/256/0/180] [ 9540.338567] LNet: Added LNI 192.168.202.156@tcp113 [8/256/0/180] [ 9540.646153] LNet: Added LNI 192.168.202.156@tcp114 [8/256/0/180] [ 9540.961001] LNet: Added LNI 192.168.202.156@tcp115 [8/256/0/180] [ 9541.275934] LNet: Added LNI 192.168.202.156@tcp116 [8/256/0/180] [ 9541.600814] LNet: Added LNI 192.168.202.156@tcp117 [8/256/0/180] [ 9541.916198] LNet: Added LNI 192.168.202.156@tcp118 [8/256/0/180] [ 9542.214568] LNet: Added LNI 192.168.202.156@tcp119 [8/256/0/180] [ 9542.510110] LNet: Added LNI 192.168.202.156@tcp120 [8/256/0/180] [ 9542.796940] LNet: Added LNI 192.168.202.156@tcp121 [8/256/0/180] [ 9543.096974] LNet: Added LNI 192.168.202.156@tcp122 [8/256/0/180] [ 9543.404809] LNet: Added LNI 192.168.202.156@tcp123 [8/256/0/180] [ 9543.715690] LNet: Added LNI 192.168.202.156@tcp124 [8/256/0/180] [ 9544.011459] LNet: Added LNI 192.168.202.156@tcp125 [8/256/0/180] [ 9544.334661] LNet: Added LNI 192.168.202.156@tcp126 [8/256/0/180] [ 9544.634352] LNet: Added LNI 192.168.202.156@tcp127 [8/256/0/180] [ 9544.929504] LNet: Added LNI 192.168.202.156@tcp128 [8/256/0/180] [ 9545.227403] LNet: Added LNI 192.168.202.156@tcp129 [8/256/0/180] [ 9545.517968] LNet: Added LNI 192.168.202.156@tcp130 [8/256/0/180] [ 9545.808081] LNet: Added LNI 192.168.202.156@tcp131 [8/256/0/180] [ 9546.106650] LNet: Added LNI 192.168.202.156@tcp132 [8/256/0/180] [ 9546.395445] LNet: Added LNI 192.168.202.156@tcp133 [8/256/0/180] [ 9546.673896] LNet: Added LNI 192.168.202.156@tcp134 [8/256/0/180] [ 9546.966572] LNet: Added LNI 192.168.202.156@tcp135 [8/256/0/180] [ 9549.186601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.193425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.245838] Lustre: 463009:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [ 9550.269253] LNetError: 460473:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9550.278124] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9550.282212] LNetError: Skipped 3 previous similar messages [ 9550.339968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9550.351248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9550.367062] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp119 [ 9550.390584] Lustre: lustre-MDT0000: new disk, initializing [ 9550.421097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.429286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9551.393178] LNetError: 460475:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9551.395908] LNetError: 460475:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 9551.399535] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9551.783132] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9553.441881] LNetError: 463037:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9553.444834] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9555.816572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9555.877222] Lustre: 463026:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9555.908261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9555.909855] Lustre: Skipped 1 previous similar message [ 9555.942074] Lustre: lustre-MDT0001: new disk, initializing [ 9555.953903] Lustre: 463827:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [ 9555.957643] Lustre: 463827:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9556.003342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.017544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9556.022108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9557.298621] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9557.535869] LNetError: 463031:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9557.538281] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9560.562828] Lustre: 464397:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [ 9560.566663] Lustre: 464397:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9560.858425] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9562.545436] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9564.775326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9564.883468] Lustre: lustre-OST0000: new disk, initializing [ 9564.885200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9564.887939] Lustre: 464999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9564.921079] Lustre: 465004:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [ 9564.924045] Lustre: 465004:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9564.943954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.728362] LNetError: 463030:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9565.731904] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9566.831829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9567.483878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9567.486945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9567.525168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9570.360835] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9572.832213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9572.833398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.834340] LustreError: Skipped 1 previous similar message [ 9572.834785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.837501] Lustre: Skipped 1 previous similar message [ 9572.841367] Lustre: Skipped 1 previous similar message [ 9577.953785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9577.956433] Lustre: Skipped 1 previous similar message [ 9578.572732] Lustre: server umount lustre-OST0000 complete [ 9581.089052] LNetError: 460474:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [ 9581.092645] LNetError: Protocol error connecting to 192.168.202.156@tcp119 at host 192.168.202.156:988: is it running a compatible version of Lustre? [ 9581.535783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9581.537686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.540754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9583.072802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9588.192461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9588.194334] Lustre: Skipped 3 previous similar messages [ 9593.312700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9593.314902] Lustre: Skipped 1 previous similar message [ 9594.335104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9594.397351] Lustre: server umount lustre-MDT0000 complete [ 9595.677246] LustreError: 463826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631454 with bad export cookie 11339791465312374737 [ 9595.680845] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9595.684057] LustreError: 463826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9595.804500] Lustre: server umount lustre-MDT0001 complete [ 9599.676713] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9600.737690] Key type lgssc unregistered [ 9600.898680] LNet: 466914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9600.901795] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9600.905153] LNet: Removed LNI 192.168.202.156@tcp [ 9601.951194] LNet: Removed LNI 192.168.202.156@tcp100 [ 9601.953185] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9601.956556] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9601.960290] LNet: Removed LNI 192.168.202.156@tcp101 [ 9602.975159] LNet: Removed LNI 192.168.202.156@tcp102 [ 9602.976461] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9602.978377] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9602.980452] LNet: Removed LNI 192.168.202.156@tcp103 [ 9603.999169] LNet: Removed LNI 192.168.202.156@tcp104 [ 9605.023225] LNet: Removed LNI 192.168.202.156@tcp105 [ 9605.025023] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9605.027124] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9605.029686] LNet: Removed LNI 192.168.202.156@tcp106 [ 9606.047181] LNet: Removed LNI 192.168.202.156@tcp107 [ 9607.071171] LNet: Removed LNI 192.168.202.156@tcp108 [ 9608.095144] LNet: Removed LNI 192.168.202.156@tcp109 [ 9609.119127] LNet: Removed LNI 192.168.202.156@tcp110 [ 9609.120479] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9609.122162] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9609.124010] LNet: Removed LNI 192.168.202.156@tcp111 [ 9610.143134] LNet: Removed LNI 192.168.202.156@tcp112 [ 9611.167123] LNet: Removed LNI 192.168.202.156@tcp113 [ 9612.191098] LNet: Removed LNI 192.168.202.156@tcp114 [ 9613.215168] LNet: Removed LNI 192.168.202.156@tcp115 [ 9614.239156] LNet: Removed LNI 192.168.202.156@tcp116 [ 9615.263113] LNet: Removed LNI 192.168.202.156@tcp117 [ 9616.287083] LNet: Removed LNI 192.168.202.156@tcp118 [ 9616.288262] LNet: Removed LNI 192.168.202.156@tcp119 [ 9617.311126] LNet: Removed LNI 192.168.202.156@tcp120 [ 9617.312295] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9617.314715] LNetError: 466914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 9617.317571] LNet: Removed LNI 192.168.202.156@tcp121 [ 9617.319444] LNet: Removed LNI 192.168.202.156@tcp122 [ 9617.320535] LNet: Removed LNI 192.168.202.156@tcp123 [ 9617.321415] LNet: Removed LNI 192.168.202.156@tcp124 [ 9617.322348] LNet: Removed LNI 192.168.202.156@tcp125 [ 9617.323281] LNet: Removed LNI 192.168.202.156@tcp126 [ 9617.324290] LNet: Removed LNI 192.168.202.156@tcp127 [ 9617.325264] LNet: Removed LNI 192.168.202.156@tcp128 [ 9617.326243] LNet: Removed LNI 192.168.202.156@tcp129 [ 9617.327806] LNet: Removed LNI 192.168.202.156@tcp130 [ 9617.329705] LNet: Removed LNI 192.168.202.156@tcp131 [ 9617.331331] LNet: Removed LNI 192.168.202.156@tcp132 [ 9617.333284] LNet: Removed LNI 192.168.202.156@tcp133 [ 9617.334994] LNet: Removed LNI 192.168.202.156@tcp134 [ 9617.342391] LNet: Removed LNI 192.168.202.156@tcp135 [ 9617.573847] Key type .llcrypt unregistered [ 9617.575037] Key type ._llcrypt unregistered [ 9624.510403] Key type ._llcrypt registered [ 9624.511656] Key type .llcrypt registered [ 9624.543179] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9630.652682] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9631.070954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.092367] alg: No test for adler32 (adler32-zlib) [ 9631.943133] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9632.029725] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9633.607149] Key type lgssc registered [ 9633.939879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9644.329467] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:38:23 (1781631503) [ 9649.998941] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9650.925704] Key type lgssc unregistered [ 9651.039281] LNet: 470972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9651.042331] LNetError: 470972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9651.050257] LNet: Removed LNI 192.168.202.156@tcp [ 9651.311466] Key type .llcrypt unregistered [ 9651.312286] Key type ._llcrypt unregistered [ 9657.729685] Key type ._llcrypt registered [ 9657.730748] Key type .llcrypt registered [ 9657.773959] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9658.193308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9658.254256] alg: No test for adler32 (adler32-zlib) [ 9659.133641] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9659.233500] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9660.823274] Key type lgssc registered [ 9661.245269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9664.332804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9664.337702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.435810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9665.446696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9665.481788] Lustre: lustre-MDT0000: new disk, initializing [ 9665.508176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.516069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9666.786942] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9670.804956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.845493] Lustre: 472159:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9670.868626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9670.870804] Lustre: Skipped 1 previous similar message [ 9670.906915] Lustre: lustre-MDT0001: new disk, initializing [ 9670.936023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9670.947116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9670.950895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9672.243704] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9675.681664] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9677.374046] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9679.446578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9679.537985] Lustre: lustre-OST0000: new disk, initializing [ 9679.539786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9679.542268] Lustre: 474131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9679.568661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.377437] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9681.658751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9681.664630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9681.706051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9684.923802] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9686.009968] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lnet_if_list [ 9687.562359] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9687.571420] LNet: Added LNI 192.168.202.156@tcp10 [8/256/0/180] [ 9689.112370] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9689.123952] LNet: Added LNI 192.168.202.156@tcp11 [8/256/0/180] [ 9690.285979] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9690.296458] LNet: Added LNI 192.168.202.156@tcp12 [8/256/0/180] [ 9691.507837] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9691.516881] LNet: Added LNI 192.168.202.156@tcp13 [8/256/0/180] [ 9692.731973] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9692.741329] LNet: Added LNI 192.168.202.156@tcp14 [8/256/0/180] [ 9694.109467] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9694.118439] LNet: Added LNI 192.168.202.156@tcp15 [8/256/0/180] [ 9695.477331] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9695.487559] LNet: Added LNI 192.168.202.156@tcp16 [8/256/0/180] [ 9696.915280] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9696.925922] LNet: Added LNI 192.168.202.156@tcp17 [8/256/0/180] [ 9698.317919] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9698.327745] LNet: Added LNI 192.168.202.156@tcp18 [8/256/0/180] [ 9699.794978] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9699.804084] LNet: Added LNI 192.168.202.156@tcp19 [8/256/0/180] [ 9701.279879] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9701.292558] LNet: Added LNI 192.168.202.156@tcp20 [8/256/0/180] [ 9702.896686] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9702.906453] LNet: Added LNI 192.168.202.156@tcp21 [8/256/0/180] [ 9704.555367] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9704.566633] LNet: Added LNI 192.168.202.156@tcp22 [8/256/0/180] [ 9706.274769] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9706.285155] LNet: Added LNI 192.168.202.156@tcp23 [8/256/0/180] [ 9707.959817] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9707.969916] LNet: Added LNI 192.168.202.156@tcp24 [8/256/0/180] [ 9709.661995] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9709.677154] LNet: Added LNI 192.168.202.156@tcp25 [8/256/0/180] [ 9711.512769] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9711.523885] LNet: Added LNI 192.168.202.156@tcp26 [8/256/0/180] [ 9713.406914] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9713.417554] LNet: Added LNI 192.168.202.156@tcp27 [8/256/0/180] [ 9715.254278] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9715.264619] LNet: Added LNI 192.168.202.156@tcp28 [8/256/0/180] [ 9717.067474] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9717.078757] LNet: Added LNI 192.168.202.156@tcp29 [8/256/0/180] [ 9718.998108] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9719.010655] LNet: Added LNI 192.168.202.156@tcp30 [8/256/0/180] [ 9720.893214] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9720.903133] LNet: Added LNI 192.168.202.156@tcp31 [8/256/0/180] [ 9722.997562] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9723.008040] LNet: Added LNI 192.168.202.156@tcp32 [8/256/0/180] [ 9724.971691] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9724.983906] LNet: Added LNI 192.168.202.156@tcp33 [8/256/0/180] [ 9727.008302] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9727.018860] LNet: Added LNI 192.168.202.156@tcp34 [8/256/0/180] [ 9728.998990] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9729.010648] LNet: Added LNI 192.168.202.156@tcp35 [8/256/0/180] [ 9730.932865] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9730.941599] LNet: Added LNI 192.168.202.156@tcp36 [8/256/0/180] [ 9732.878807] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9732.890613] LNet: Added LNI 192.168.202.156@tcp37 [8/256/0/180] [ 9734.895568] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9734.905391] LNet: Added LNI 192.168.202.156@tcp38 [8/256/0/180] [ 9736.951333] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9736.964064] LNet: Added LNI 192.168.202.156@tcp39 [8/256/0/180] [ 9739.179381] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9739.191752] LNet: Added LNI 192.168.202.156@tcp40 [8/256/0/180] [ 9741.414348] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9741.425141] LNet: Added LNI 192.168.202.156@tcp41 [8/256/0/180] [ 9743.665279] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9743.673933] LNet: Added LNI 192.168.202.156@tcp42 [8/256/0/180] [ 9745.935504] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9745.945192] LNet: Added LNI 192.168.202.156@tcp43 [8/256/0/180] [ 9748.218188] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9748.228904] LNet: Added LNI 192.168.202.156@tcp44 [8/256/0/180] [ 9750.533505] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9750.544406] LNet: Added LNI 192.168.202.156@tcp45 [8/256/0/180] [ 9752.805443] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9752.815619] LNet: Added LNI 192.168.202.156@tcp46 [8/256/0/180] [ 9755.116069] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9755.125900] LNet: Added LNI 192.168.202.156@tcp47 [8/256/0/180] [ 9757.568177] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9757.578947] LNet: Added LNI 192.168.202.156@tcp48 [8/256/0/180] [ 9760.131335] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9760.144414] LNet: Added LNI 192.168.202.156@tcp49 [8/256/0/180] [ 9762.779631] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9762.788840] LNet: Added LNI 192.168.202.156@tcp50 [8/256/0/180] [ 9773.535813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9773.537755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.541752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9774.048274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9774.049397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.050327] Lustre: Skipped 1 previous similar message [ 9774.052800] Lustre: Skipped 1 previous similar message [ 9775.895923] Lustre: server umount lustre-MDT0000 complete [ 9777.060048] LustreError: 472154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631636 with bad export cookie 5753087078375591388 [ 9777.063085] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9777.063885] LustreError: 472154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9777.163765] Lustre: server umount lustre-MDT0001 complete [ 9784.250919] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9786.958050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9787.295339] LustreError: 471434:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000650eb6f0 x1868176050699008/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9787.405832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.646467] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9791.274160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.357817] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.579906] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9794.467384] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9794.728769] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9795.487074] Lustre: 471436:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781631638/real 1781631638] req@00000000b1dc7e9f x1868176050698624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781631654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9796.353814] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9798.623162] Lustre: 471437:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781631641/real 1781631641] req@000000001b89b897 x1868176050698752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781631657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9800.671151] Lustre: 471438:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781631643/real 1781631643] req@0000000029acfd22 x1868176050698880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781631659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9804.770270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9804.771175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9804.773500] Lustre: Skipped 1 previous similar message [ 9809.232769] Lustre: server umount lustre-OST0000 complete [ 9811.935813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9811.938040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.942526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.945031] Lustre: Skipped 1 previous similar message [ 9816.527995] Lustre: server umount lustre-MDT0000 complete [ 9817.666520] LustreError: 472957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631676 with bad export cookie 5753087078375593432 [ 9817.668265] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9817.670117] LustreError: 472957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9817.782360] Lustre: server umount lustre-MDT0001 complete [ 9821.121397] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9822.069842] Key type lgssc unregistered [ 9822.228745] LNet: 501085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9822.231900] LNetError: 501085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9822.233870] LNet: Removed LNI 192.168.202.156@tcp [ 9822.234832] LNet: Removed LNI 192.168.202.156@tcp10 [ 9822.235851] LNet: Removed LNI 192.168.202.156@tcp11 [ 9822.236838] LNet: Removed LNI 192.168.202.156@tcp12 [ 9822.237783] LNet: Removed LNI 192.168.202.156@tcp13 [ 9822.239187] LNet: Removed LNI 192.168.202.156@tcp14 [ 9822.240131] LNet: Removed LNI 192.168.202.156@tcp15 [ 9822.241075] LNet: Removed LNI 192.168.202.156@tcp16 [ 9822.241951] LNet: Removed LNI 192.168.202.156@tcp17 [ 9822.243211] LNet: Removed LNI 192.168.202.156@tcp18 [ 9822.244544] LNet: Removed LNI 192.168.202.156@tcp19 [ 9822.245937] LNet: Removed LNI 192.168.202.156@tcp20 [ 9822.247077] LNet: Removed LNI 192.168.202.156@tcp21 [ 9822.247901] LNet: Removed LNI 192.168.202.156@tcp22 [ 9822.248866] LNet: Removed LNI 192.168.202.156@tcp23 [ 9822.249852] LNet: Removed LNI 192.168.202.156@tcp24 [ 9822.250797] LNet: Removed LNI 192.168.202.156@tcp25 [ 9822.251811] LNet: Removed LNI 192.168.202.156@tcp26 [ 9822.252740] LNet: Removed LNI 192.168.202.156@tcp27 [ 9822.253671] LNet: Removed LNI 192.168.202.156@tcp28 [ 9822.254583] LNet: Removed LNI 192.168.202.156@tcp29 [ 9822.255568] LNet: Removed LNI 192.168.202.156@tcp30 [ 9822.256476] LNet: Removed LNI 192.168.202.156@tcp31 [ 9822.257340] LNet: Removed LNI 192.168.202.156@tcp32 [ 9822.258306] LNet: Removed LNI 192.168.202.156@tcp33 [ 9822.259234] LNet: Removed LNI 192.168.202.156@tcp34 [ 9822.260327] LNet: Removed LNI 192.168.202.156@tcp35 [ 9822.261483] LNet: Removed LNI 192.168.202.156@tcp36 [ 9822.262448] LNet: Removed LNI 192.168.202.156@tcp37 [ 9822.263390] LNet: Removed LNI 192.168.202.156@tcp38 [ 9822.264370] LNet: Removed LNI 192.168.202.156@tcp39 [ 9822.265348] LNet: Removed LNI 192.168.202.156@tcp40 [ 9822.266393] LNet: Removed LNI 192.168.202.156@tcp41 [ 9822.267665] LNet: Removed LNI 192.168.202.156@tcp42 [ 9822.268673] LNet: Removed LNI 192.168.202.156@tcp43 [ 9822.269682] LNet: Removed LNI 192.168.202.156@tcp44 [ 9822.271447] LNet: Removed LNI 192.168.202.156@tcp45 [ 9822.273349] LNet: Removed LNI 192.168.202.156@tcp46 [ 9822.274841] LNet: Removed LNI 192.168.202.156@tcp47 [ 9822.275842] LNet: Removed LNI 192.168.202.156@tcp48 [ 9822.276797] LNet: Removed LNI 192.168.202.156@tcp49 [ 9822.281400] LNet: Removed LNI 192.168.202.156@tcp50 [ 9822.542585] Key type .llcrypt unregistered [ 9822.543518] Key type ._llcrypt unregistered [ 9828.577590] Key type ._llcrypt registered [ 9828.578615] Key type .llcrypt registered [ 9828.612496] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9834.405657] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9834.823105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9834.838575] alg: No test for adler32 (adler32-zlib) [ 9835.684074] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9835.763333] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9837.343196] Key type lgssc registered [ 9837.706451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9848.846505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9848.853663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.943125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9849.956069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9849.989682] Lustre: lustre-MDT0000: new disk, initializing [ 9850.010817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9850.016516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9851.269642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9855.314223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.357525] Lustre: 504255:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9855.394526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9855.395986] Lustre: Skipped 1 previous similar message [ 9855.433717] Lustre: lustre-MDT0001: new disk, initializing [ 9855.462858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9855.474214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9855.476619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9856.708780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9860.158945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9861.878638] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9864.303206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9864.380359] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9864.405118] Lustre: lustre-OST0000: new disk, initializing [ 9864.406689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9864.408952] Lustre: 506272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9864.437195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.297590] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9869.936498] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9870.327964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9870.332398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9870.375901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9875.424885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9875.426914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.427710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9875.430474] Lustre: Skipped 1 previous similar message [ 9878.290620] Lustre: server umount lustre-OST0000 complete [ 9880.544731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.545370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9880.552241] Lustre: Skipped 1 previous similar message [ 9880.554080] Lustre: Skipped 1 previous similar message [ 9885.661048] Lustre: server umount lustre-MDT0000 complete [ 9885.664652] LustreError: 504266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9885.669558] LustreError: 504266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9886.980128] LustreError: 504247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631746 with bad export cookie 11259562605689330370 [ 9886.983697] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9886.985406] LustreError: 504247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9887.104240] Lustre: server umount lustre-MDT0001 complete [ 9890.888130] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9891.997743] Key type lgssc unregistered [ 9892.124355] LNet: 508176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9892.127833] LNetError: 508176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9892.136267] LNet: Removed LNI 192.168.202.156@tcp [ 9892.396461] Key type .llcrypt unregistered [ 9892.397312] Key type ._llcrypt unregistered [ 9899.421475] Key type ._llcrypt registered [ 9899.422526] Key type .llcrypt registered [ 9899.461055] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9899.786191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9899.827387] alg: No test for adler32 (adler32-zlib) [ 9900.675612] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9900.757921] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9902.335149] Key type lgssc registered [ 9902.667692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9905.079397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9905.084593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.224270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.442603] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9910.042866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.146251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9911.453789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9913.921025] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9915.761502] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9918.127104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.251968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9920.332958] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9922.881744] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9936.943199] loop0: detected capacity change from 0 to 204800000 [ 9937.005332] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9937.010205] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9937.016930] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9938.853478] loop0: detected capacity change from 0 to 204800000 [ 9938.918389] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9938.921949] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9938.925527] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9940.473139] loop0: detected capacity change from 0 to 204800000 [ 9942.342897] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9942.392112] Lustre: fs15246-OST0000: new disk, initializing [ 9942.393914] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9942.396599] Lustre: 513034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9942.417579] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.372567] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9948.044296] loop1: detected capacity change from 0 to 204800000 [ 9949.812802] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.847879] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9949.872346] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9949.911170] Lustre: fs15246-MDT0000: new disk, initializing [ 9949.962463] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9949.970121] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9950.009909] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9951.072285] Lustre: 514155:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000cb7d2833 x1868176303357568/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781631826 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9951.270572] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9960.159307] LustreError: 514155:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9960.162729] LustreError: 514155:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9960.165720] LustreError: 514155:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9962.467593] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.476381] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.493281] Lustre: server umount fs15246-MDT0000 complete [ 9967.412937] Lustre: server umount fs15246-OST0000 complete [ 9971.680610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9971.683031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.686916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9972.704650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9972.706307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.706650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9976.783078] Lustre: server umount lustre-OST0000 complete [ 9981.920028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9981.922525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9981.928855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9981.930939] Lustre: Skipped 1 previous similar message [ 9984.283815] Lustre: server umount lustre-MDT0000 complete [ 9985.578614] LustreError: 514149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631844 with bad export cookie 4366055400619346402 [ 9985.581974] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9985.582120] LustreError: 514149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9985.721103] Lustre: server umount lustre-MDT0001 complete [ 9989.585442] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9990.706756] Key type lgssc unregistered [ 9990.836487] LNet: 517249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9990.840369] LNetError: 517249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9990.852494] LNet: Removed LNI 192.168.202.156@tcp [ 9991.179340] Key type .llcrypt unregistered [ 9991.180443] Key type ._llcrypt unregistered [ 9998.161662] Key type ._llcrypt registered [ 9998.162660] Key type .llcrypt registered [ 9998.199406] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9998.552363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9998.563605] alg: No test for adler32 (adler32-zlib) [ 9999.429517] Lustre: Lustre: Build Version: 2.17.54_40_g15f0a2e [ 9999.508172] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10001.087170] Key type lgssc registered [10001.496817] Lustre: Echo OBD driver; http://www.lustre.org/ [10014.197793] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10017.541309] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10028.740789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10028.745919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.826508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10029.837735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10029.872808] Lustre: lustre-MDT0000: new disk, initializing [10029.895629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.901342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10031.205404] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10035.207700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.249325] Lustre: 521685:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10035.274310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10035.275812] Lustre: Skipped 1 previous similar message [10035.311532] Lustre: lustre-MDT0001: new disk, initializing [10035.346936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10035.360248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10035.365069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10036.622368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10040.156233] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10041.932435] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10044.003680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10044.094166] Lustre: lustre-OST0000: new disk, initializing [10044.095657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10044.097947] Lustre: 523650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10044.122520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10045.927342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10049.478068] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10050.749644] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10051.061066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10051.063899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10051.092435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10051.836063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10053.218589] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10053.290857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10056.162627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10056.163517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.171097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10056.173371] Lustre: Skipped 1 previous similar message [10060.056582] Lustre: server umount lustre-OST0000 complete [10066.400859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10066.401220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.401975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.403252] LustreError: Skipped 1 previous similar message [10066.409053] Lustre: Skipped 2 previous similar messages [10067.487277] Lustre: server umount lustre-MDT0000 complete [10068.756329] LustreError: 521677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781631927 with bad export cookie 6447753298559139111 [10068.760309] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.761048] LustreError: 521677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10068.883723] Lustre: server umount lustre-MDT0001 complete [10077.027216] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10080.543206] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10090.539769] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9827 sec ========== 13:45:49 (1781631949) [10091.104130] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:45:49 (1781631949) === [10092.198388] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:45:51 (1781631951) === [10105.106355] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10106.171167] Key type lgssc unregistered [10106.305305] LNet: 530288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10106.308351] LNetError: 530288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10106.318323] LNet: Removed LNI 192.168.202.156@tcp [10106.633095] Key type .llcrypt unregistered [10106.634136] Key type ._llcrypt unregistered