[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518309615 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.012090] *** VALIDATE selinux *** [ 0.021116] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026300] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029029] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032226] *** VALIDATE proc *** [ 0.033244] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.043327] debug: unmapping init [mem 0xffffffffb0459000-0xffffffffb0460fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053280] rcu: Hierarchical SRCU implementation. [ 0.055489] smp: Bringing up secondary CPUs ... [ 0.056590] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.061123] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233000] node 0 deferred pages initialised in 168ms [ 0.238237] devtmpfs: initialized [ 0.239246] x86/mm: Memory block size: 128MB [ 0.241811] gcov: version magic: 0x41383552 [ 0.243508] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246153] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248398] pinctrl core: initialized pinctrl subsystem [ 0.250199] [ 0.250660] ************************************************************* [ 0.252017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254019] ** ** [ 0.257016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.259017] ** ** [ 0.261016] ** This means that this kernel is built to expose internal ** [ 0.263017] ** IOMMU data structures, which may compromise security on ** [ 0.265018] ** your system. ** [ 0.268018] ** ** [ 0.270015] ** If you see this message and you are not debugging the ** [ 0.272019] ** kernel, report this immediately to your vendor! ** [ 0.274020] ** ** [ 0.277016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279013] ************************************************************* [ 0.281869] NET: Registered protocol family 16 [ 0.283518] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.286085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.292200] cpuidle: using governor menu [ 0.294000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.297197] PCI: Using configuration type 1 for base access [ 0.300164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309163] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312107] cryptd: max_cpu_qlen set to 1000 [ 0.315372] ACPI: Added _OSI(Module Device) [ 0.317021] ACPI: Added _OSI(Processor Device) [ 0.318015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.320017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325860] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332632] ACPI: Interpreter enabled [ 0.334070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.335013] ACPI: Using IOAPIC for interrupt routing [ 0.336137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.338309] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347559] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.349048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360694] acpiphp: Slot [2] registered [ 0.362243] acpiphp: Slot [5] registered [ 0.364264] acpiphp: Slot [6] registered [ 0.366170] acpiphp: Slot [7] registered [ 0.367175] acpiphp: Slot [8] registered [ 0.369192] acpiphp: Slot [9] registered [ 0.371180] acpiphp: Slot [10] registered [ 0.372193] acpiphp: Slot [3] registered [ 0.374157] acpiphp: Slot [4] registered [ 0.376164] acpiphp: Slot [11] registered [ 0.378190] acpiphp: Slot [12] registered [ 0.380186] acpiphp: Slot [13] registered [ 0.381084] acpiphp: Slot [14] registered [ 0.384214] acpiphp: Slot [15] registered [ 0.386140] acpiphp: Slot [16] registered [ 0.388151] acpiphp: Slot [17] registered [ 0.389174] acpiphp: Slot [18] registered [ 0.391137] acpiphp: Slot [19] registered [ 0.392183] acpiphp: Slot [20] registered [ 0.394163] acpiphp: Slot [21] registered [ 0.395154] acpiphp: Slot [22] registered [ 0.397149] acpiphp: Slot [23] registered [ 0.399145] acpiphp: Slot [24] registered [ 0.400141] acpiphp: Slot [25] registered [ 0.402090] acpiphp: Slot [26] registered [ 0.403097] acpiphp: Slot [27] registered [ 0.404136] acpiphp: Slot [28] registered [ 0.406135] acpiphp: Slot [29] registered [ 0.407166] acpiphp: Slot [30] registered [ 0.408455] acpiphp: Slot [31] registered [ 0.410116] PCI host bridge to bus 0000:00 [ 0.412035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.413024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.415035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.418034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.420038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.423064] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.425236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427122] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.431303] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.444953] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.448028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456420] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.458748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464808] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.469022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.487023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.492946] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.499026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.508024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.524023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.535731] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.545022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.552020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.573030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.583075] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.590021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.597016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.617016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.633444] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.649013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.670015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.688725] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.696015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.702015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.719019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.732657] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.738019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.743030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.758036] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.767912] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.771417] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.773350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.776374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.778251] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.783207] iommu: Default domain type: Passthrough [ 0.784427] SCSI subsystem initialized [ 0.785000] ACPI: bus type USB registered [ 0.787093] usbcore: registered new interface driver usbfs [ 0.789078] usbcore: registered new interface driver hub [ 0.791135] usbcore: registered new device driver usb [ 0.793146] pps_core: LinuxPPS API ver. 1 registered [ 0.794023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.798097] PTP clock support registered [ 0.800101] EDAC MC: Ver: 3.0.0 [ 0.802007] PCI: Using ACPI for IRQ routing [ 0.803683] NetLabel: Initializing [ 0.805016] NetLabel: domain hash size = 128 [ 0.807013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.809279] NetLabel: unlabeled traffic allowed by default [ 0.812139] vgaarb: loaded [ 0.814440] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.816011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.823145] clocksource: Switched to clocksource kvm-clock [ 0.946197] VFS: Disk quotas dquot_6.6.0 [ 0.947928] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.951285] *** VALIDATE ramfs *** [ 0.952536] *** VALIDATE hugetlbfs *** [ 0.954346] pnp: PnP ACPI init [ 0.957502] pnp: PnP ACPI: found 6 devices [ 0.975820] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.979498] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.981994] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.984481] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.987224] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.990029] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.992927] NET: Registered protocol family 2 [ 0.995985] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.001243] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.004624] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.009117] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012876] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015760] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.018700] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.021801] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024796] NET: Registered protocol family 1 [ 1.028234] RPC: Registered named UNIX socket transport module. [ 1.031777] RPC: Registered udp transport module. [ 1.033589] RPC: Registered tcp transport module. [ 1.035356] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038147] NET: Registered protocol family 44 [ 1.039768] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.041357] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.043369] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.045876] PCI: CLS 0 bytes, default 64 [ 1.047647] Unpacking initramfs... [ 2.452650] debug: unmapping init [mem 0xffff9c5c3cc54000-0xffff9c5c3ffbffff] [ 2.456712] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.459072] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.462174] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.976616] Initialise system trusted keyrings [ 2.978445] Key type blacklist registered [ 2.981774] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990629] zbud: loaded [ 2.993795] *** VALIDATE nfs *** [ 2.995150] *** VALIDATE nfs4 *** [ 2.996981] pstore: using deflate compression [ 3.000797] Platform Keyring initialized [ 3.086472] NET: Registered protocol family 38 [ 3.089132] Key type asymmetric registered [ 3.091456] Asymmetric key parser 'x509' registered [ 3.093863] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.097815] io scheduler mq-deadline registered [ 3.099736] io scheduler kyber registered [ 3.101791] io scheduler bfq registered [ 3.104329] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107382] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.110220] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.113651] ACPI: Power Button [PWRF] [ 3.119718] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.126570] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.138976] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.145281] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.157194] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.191793] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.222826] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.228656] Non-volatile memory driver v1.3 [ 3.230384] Linux agpgart interface v0.103 [ 3.266471] virtio_blk virtio1: [vda] 141800 512-byte logical blocks (72.6 MB/69.2 MiB) [ 3.270013] vda: detected capacity change from 0 to 72601600 [ 3.287187] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.290151] vdb: detected capacity change from 0 to 1073741824 [ 3.309464] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.313247] vdc: detected capacity change from 0 to 2621440000 [ 3.337453] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.341121] vdd: detected capacity change from 0 to 2621440000 [ 3.365703] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.369434] vde: detected capacity change from 0 to 4294967296 [ 3.391789] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.395125] vdf: detected capacity change from 0 to 4294967296 [ 3.402373] libphy: Fixed MDIO Bus: probed [ 3.406848] usbcore: registered new interface driver usbserial_generic [ 3.408881] usbserial: USB Serial support registered for generic [ 3.410724] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.414197] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.415533] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.417651] mousedev: PS/2 mouse device common for all mice [ 3.420187] rtc_cmos 00:05: RTC can wake from S4 [ 3.426523] rtc_cmos 00:05: registered as rtc0 [ 3.428534] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.431192] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.433000] intel_pstate: CPU model not supported [ 3.442831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.442885] hid: raw HID events driver (C) Jiri Kosina [ 3.448567] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.452139] usbcore: registered new interface driver usbhid [ 3.453852] usbhid: USB HID core driver [ 3.455400] drop_monitor: Initializing network drop monitor service [ 3.457924] Initializing XFRM netlink socket [ 3.460143] NET: Registered protocol family 10 [ 3.464751] Segment Routing with IPv6 [ 3.465958] NET: Registered protocol family 17 [ 3.468404] mpls_gso: MPLS GSO support [ 3.474361] RAS: Correctable Errors collector initialized. [ 3.476777] AVX version of gcm_enc/dec engaged. [ 3.478700] AES CTR mode by8 optimization enabled [ 3.564665] sched_clock: Marking stable (3564560116, 0)->(4539686883, -975126767) [ 3.569110] registered taskstats version 1 [ 3.571416] Loading compiled-in X.509 certificates [ 3.573335] zswap: loaded using pool lzo/zbud [ 3.596328] Key type big_key registered [ 3.610288] Key type encrypted registered [ 3.611828] ima: No TPM chip found, activating TPM-bypass! [ 3.614254] ima: Allocated hash algorithm: sha1 [ 3.616333] ima: No architecture policies found [ 3.618425] evm: Initialising EVM extended attributes: [ 3.620644] evm: security.selinux [ 3.622120] evm: security.ima [ 3.623492] evm: security.capability [ 3.625104] evm: HMAC attrs: 0x1 [ 3.627840] rtc_cmos 00:05: setting system clock to 2026-05-18 17:47:09 UTC (1779126429) [ 3.634665] debug: unmapping init [mem 0xffffffffb1403000-0xffffffffb15fffff] [ 3.638162] debug: unmapping init [mem 0xffffffffb0182000-0xffffffffb0458fff] [ 3.647132] Write protecting the kernel read-only data: 28672k [ 3.650800] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 3.653962] debug: unmapping init [mem 0xffffffffaf114000-0xffffffffaf1fffff] [ 3.690410] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.699770] systemd[1]: Detected virtualization kvm. [ 3.704095] systemd[1]: Detected architecture x86-64. [ 3.706276] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.737872] systemd[1]: No hostname configured. [ 3.740660] systemd[1]: Set hostname to . [ 3.743342] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746859] systemd[1]: Initializing machine ID from random generator. [ 3.809433] random: ln: uninitialized urandom read (6 bytes read) [ 3.912182] random: systemd: uninitialized urandom read (16 bytes read) [ 3.915518] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.921076] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.931276] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.624390] device-mapper: uevent: version 1.0.3 [ 4.626821] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.393572] virtio_net virtio0 ens2: renamed from eth0 [ 5.424357] random: fast init done [ 5.473548] scsi host0: ata_piix [ 5.527033] scsi host1: ata_piix [ 5.529030] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.531758] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.718170] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.133437] random: crng init done [ 10.134772] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.524639] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.655918] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.904097] SELinux: Disabled at runtime. [ 11.968439] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.977130] systemd[1]: Detected virtualization kvm. [ 11.979152] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.432257] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.436733] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.442205] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.447284] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.450747] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.460722] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.467447] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK [[ 12.580854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.908530] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.253746] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.301822] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.381474] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.400246] EDAC sbridge: Ver: 1.1.2 [ 14.955484] Key type dns_resolver registered [ 15.266429] NFS: Registering the id_resolver key type [ 15.268674] Key type id_resolver registered [ 15.270570] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg643-server login: [ 30.703476] hrtimer: interrupt took 2459863 ns [ 61.365897] libcfs: loading out-of-tree module taints kernel. [ 61.403792] Key type ._llcrypt registered [ 61.406559] Key type .llcrypt registered [ 61.487743] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 79.335533] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 80.720895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.741385] alg: No test for adler32 (adler32-zlib) [ 82.030223] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 83.008080] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 84.898187] Key type lgssc registered [ 86.667917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.170996] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.408655] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 160.200689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.224070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.526042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.562857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.662765] Lustre: lustre-MDT0000: new disk, initializing [ 161.793171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.811194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.695397] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.694848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.769211] Lustre: 6544:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.794842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.800164] Lustre: Skipped 1 previous similar message [ 179.886356] Lustre: lustre-MDT0001: new disk, initializing [ 179.945111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.985271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.007248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.053328] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.142884] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.428364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.651933] Lustre: lustre-OST0000: new disk, initializing [ 196.655216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.659913] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.714327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.915674] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.278831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.287557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.368668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.858546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.973113] Lustre: lustre-OST0001: new disk, initializing [ 216.977436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.982520] Lustre: 9503:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.049341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.064709] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.819211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.831379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.898571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.396841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.982111] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.691281] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.182913] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing yml_node [ 256.124505] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 259.236980] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 262.899626] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 264.607831] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 18 13:51:28 EDT 2026 [ 283.101190] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 284.502032] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.029258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.041697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.051331] Lustre: Skipped 2 previous similar messages [ 301.059861] Lustre: Skipped 2 previous similar messages [ 306.148120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.161600] Lustre: Skipped 3 previous similar messages [ 306.632614] Lustre: server umount lustre-MDT0000 complete [ 309.923941] LustreError: 9504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779126735 with bad export cookie 10845590034832318889 [ 309.927161] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.940188] LustreError: 9504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 311.271624] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.293262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.298969] Lustre: Skipped 2 previous similar messages [ 311.313506] Lustre: Skipped 1 previous similar message [ 314.273967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.279326] Lustre: Skipped 1 previous similar message [ 316.361810] Lustre: server umount lustre-MDT0001 complete [ 320.206276] Lustre: server umount lustre-OST0000 complete [ 323.928217] Lustre: server umount lustre-OST0001 complete [ 329.788064] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 338.688608] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 373.317513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.542675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 373.586726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 373.655192] Lustre: lustre-MDT0000: new disk, initializing [ 373.759408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.796057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 377.431425] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 387.773416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.898914] Lustre: 15025:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.946697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.950385] Lustre: Skipped 1 previous similar message [ 388.003823] Lustre: lustre-MDT0001: new disk, initializing [ 388.152371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.195505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.213638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.787529] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 401.749752] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 406.684134] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.337427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.756599] Lustre: lustre-OST0000: new disk, initializing [ 414.774923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.788883] Lustre: 16768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 414.889589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.116081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.129743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.259692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.499475] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 429.284494] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 433.292730] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.501331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.168180] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 438.374892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.832123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.832918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.853407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.865878] Lustre: Skipped 1 previous similar message [ 452.068375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.087270] Lustre: Skipped 3 previous similar messages [ 455.136686] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 455.407592] Lustre: server umount lustre-OST0000 complete [ 459.746369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.760433] LustreError: Skipped 1 previous similar message [ 459.769828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.786403] Lustre: Skipped 1 previous similar message [ 465.931939] Lustre: server umount lustre-MDT0000 complete [ 467.447508] LustreError: 15038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.477925] LustreError: 15038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 470.073235] LustreError: 15018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779126895 with bad export cookie 10845590034832320422 [ 470.082698] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.752182] Lustre: server umount lustre-MDT0001 complete [ 475.735295] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:54:59 (1779126899) [ 478.479423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 480.126484] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:55:04 (1779126904) [ 486.941555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.431657] LustreError: 19031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.519692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.365466] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 492.520542] LustreError: 19032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.637440] LustreError: 19031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.494496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.142692] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 509.829793] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.803890] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 519.965713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 526.997328] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 532.901993] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 543.212278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.233620] Lustre: Skipped 2 previous similar messages [ 543.237826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 543.238129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.252089] Lustre: Skipped 3 previous similar messages [ 547.256582] Lustre: server umount lustre-OST0000 complete [ 551.397331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.411830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.091855] Lustre: server umount lustre-MDT0000 complete [ 558.561160] LustreError: 20354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.585711] LustreError: 20354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 560.672485] LustreError: 19011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779126986 with bad export cookie 10845590034832321304 [ 560.683914] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.692836] LustreError: 19011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 560.955648] Lustre: server umount lustre-MDT0001 complete [ 571.334817] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 574.181438] Key type lgssc unregistered [ 574.482106] LNet: 22205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.489259] LNetError: 22205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.526344] LNet: Removed LNI 192.168.206.143@tcp [ 575.617264] Key type .llcrypt unregistered [ 575.619940] Key type ._llcrypt unregistered [ 601.107891] Key type ._llcrypt registered [ 601.110164] Key type .llcrypt registered [ 601.196960] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 602.438106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.459147] alg: No test for adler32 (adler32-zlib) [ 603.535558] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 603.818717] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 605.504202] Key type lgssc registered [ 606.723828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.194996] loop: module loaded [ 615.264439] loop0: detected capacity change from 0 to 204800000 [ 615.475830] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.489054] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.892714] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.852520] loop0: detected capacity change from 0 to 204800000 [ 621.990688] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.000814] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.030202] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.332588] loop0: detected capacity change from 0 to 204800000 [ 633.037066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 633.074807] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.312841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 634.358951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 634.426262] Lustre: lustre-MDT0000: new disk, initializing [ 634.540759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.562858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 639.517664] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 648.717341] loop1: detected capacity change from 0 to 204800000 [ 655.163230] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 655.415497] Lustre: lustre-OST07c6: new disk, initializing [ 655.421147] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 655.431560] Lustre: Skipped 1 previous similar message [ 655.448266] Lustre: 25413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 655.588048] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 661.212148] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 665.136648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 665.147460] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 665.175610] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 667.052726] Lustre: 24413:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 667.081160] Lustre: 24413:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 667.102705] Lustre: 24413:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 667.119035] Lustre: 24413:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 667.134084] Lustre: 24413:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 667.143689] Lustre: 24413:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 681.170485] Lustre: 25913:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 681.186622] Lustre: 25913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 681.195524] Lustre: 25913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.207391] Lustre: 25913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 681.225605] Lustre: 25913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 681.231650] Lustre: 25913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.267889] Lustre: 24413:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 686.276941] Lustre: 24413:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.281301] Lustre: 24413:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 686.285989] Lustre: 24413:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.294589] Lustre: 24413:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.299832] Lustre: 24413:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.303855] Lustre: 24413:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 686.310782] Lustre: 24413:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.316137] Lustre: 24413:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 686.320959] Lustre: 24413:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.325538] Lustre: 24413:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.330531] Lustre: 24413:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.564247] Lustre: 24415:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 689.569154] Lustre: 24415:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.572673] Lustre: 24415:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 689.578338] Lustre: 24415:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.582994] Lustre: 24415:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.591147] Lustre: 24415:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.595459] Lustre: 24415:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 689.601058] Lustre: 24415:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.604279] Lustre: 24415:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 689.607929] Lustre: 24415:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.610097] Lustre: 24415:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.613675] Lustre: 24415:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.367421] Lustre: 24415:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 694.384108] Lustre: 24415:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 694.400528] Lustre: 24415:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.418694] Lustre: 24415:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.429045] Lustre: 24415:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.442827] Lustre: 24415:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.454374] Lustre: 24415:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 694.465863] Lustre: 24415:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.474953] Lustre: 24415:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 694.486239] Lustre: 24415:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.497381] Lustre: 24415:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.511044] Lustre: 24415:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.619617] Lustre: 25913:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 705.633422] Lustre: 25913:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 705.644105] Lustre: 25913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 705.649873] Lustre: 25913:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.654639] Lustre: 25913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.663756] Lustre: 25913:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.672884] Lustre: 25913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 705.683253] Lustre: 25913:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.695231] Lustre: 25913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 705.703817] Lustre: 25913:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.711198] Lustre: 25913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.720869] Lustre: 25913:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.098879] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 752.100241] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.123245] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.217482] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.897602] Lustre: server umount lustre-OST07c6 complete [ 766.178953] Lustre: server umount lustre-MDT0000 complete [ 779.829862] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 782.395636] Key type lgssc unregistered [ 782.704087] LNet: 27935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.710440] LNetError: 27935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.729783] LNet: Removed LNI 192.168.206.143@tcp [ 783.594064] Key type .llcrypt unregistered [ 783.598109] Key type ._llcrypt unregistered [ 805.576304] Key type ._llcrypt registered [ 805.579670] Key type .llcrypt registered [ 805.659740] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 806.412440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.434911] alg: No test for adler32 (adler32-zlib) [ 807.791395] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 808.094492] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 809.752166] Key type lgssc registered [ 810.944823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.201485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.222198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.860509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.149069] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 830.436066] LustreError: 29257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.465720] LustreError: 29257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 834.015310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.483196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.249623] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 844.355229] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 847.465769] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 853.410933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.810888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.225858] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 864.526977] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.110393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 880.615320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.630567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.648613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.634911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.642284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.648580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.538985] Lustre: server umount lustre-OST0000 complete [ 886.760725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.775126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 886.792688] Lustre: Skipped 2 previous similar messages [ 891.879577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.892038] Lustre: Skipped 1 previous similar message [ 892.659585] Lustre: server umount lustre-MDT0000 complete [ 895.719876] LustreError: 29236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779127321 with bad export cookie 3650034884680022685 [ 895.732472] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.971580] Lustre: server umount lustre-MDT0001 complete [ 906.199306] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 909.438106] Key type lgssc unregistered [ 909.789922] LNet: 32649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.796806] LNetError: 32649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.814359] LNet: Removed LNI 192.168.206.143@tcp [ 910.533578] Key type .llcrypt unregistered [ 910.535288] Key type ._llcrypt unregistered [ 931.816842] Key type ._llcrypt registered [ 931.820136] Key type .llcrypt registered [ 931.945707] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 933.383234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.407096] alg: No test for adler32 (adler32-zlib) [ 934.569118] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 934.757051] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 936.408206] Key type lgssc registered [ 937.517033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.523415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.537793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.078261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.704597] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 956.386583] LustreError: 33955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.409000] LustreError: 33955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 960.470535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.854069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 965.583935] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 970.919471] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 974.911924] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.449390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.746484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.355728] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 992.014956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 995.025920] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1006.564554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.569453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.584406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.586429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.594717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.601619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.630392] Lustre: Skipped 1 previous similar message [ 1012.575528] Lustre: server umount lustre-OST0000 complete [ 1017.315494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.325781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.354169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.947260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.960519] Lustre: Skipped 2 previous similar messages [ 1023.564877] Lustre: server umount lustre-MDT0000 complete [ 1028.010132] LustreError: 36196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779127453 with bad export cookie 17475495527847332829 [ 1028.017533] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.029147] LustreError: 36196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1028.677973] Lustre: server umount lustre-MDT0001 complete [ 1042.363423] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1046.926146] Key type lgssc unregistered [ 1047.396829] LNet: 37133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.404697] LNetError: 37133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.431669] LNet: Removed LNI 192.168.206.143@tcp [ 1048.935042] Key type .llcrypt unregistered [ 1048.938186] Key type ._llcrypt unregistered [ 1073.855424] Key type ._llcrypt registered [ 1073.857993] Key type .llcrypt registered [ 1073.983583] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1074.809184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.935730] alg: No test for adler32 (adler32-zlib) [ 1075.929621] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1076.086227] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1077.728212] Key type lgssc registered [ 1078.956608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.061543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.091549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.696161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.771579] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1098.209466] LustreError: 38587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.227126] LustreError: 38587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1101.247409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.530978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.790033] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1111.800856] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1115.803376] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1122.208060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.568161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.738530] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1133.878766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1135.791343] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1142.619459] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 1142.755665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.771324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.290942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.291970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.302766] Lustre: Skipped 2 previous similar messages [ 1144.319553] Lustre: Skipped 1 previous similar message [ 1147.717133] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 1148.213396] Lustre: server umount lustre-MDT0000 complete [ 1149.425513] LustreError: 38588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.475889] LustreError: 38588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1152.663071] LustreError: 38569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779127578 with bad export cookie 1780916052849980131 [ 1152.670252] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.677185] LustreError: 38569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1152.823465] LustreError: 38584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.872912] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [ 1154.535647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.537347] LustreError: 38582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.537541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1157.571342] LustreError: 38582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1157.592940] Lustre: Skipped 1 previous similar message [ 1159.140298] Lustre: server umount lustre-MDT0001 complete [ 1188.320198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1188.521440] Lustre: server umount lustre-OST0000 complete [ 1204.765484] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1207.516339] Key type lgssc unregistered [ 1207.695598] LNet: 42090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.703254] LNetError: 42090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.723125] LNet: Removed LNI 192.168.206.143@tcp [ 1208.474045] Key type .llcrypt unregistered [ 1208.479221] Key type ._llcrypt unregistered [ 1235.669458] Key type ._llcrypt registered [ 1235.675587] Key type .llcrypt registered [ 1235.846918] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1237.123797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.259745] alg: No test for adler32 (adler32-zlib) [ 1238.381298] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1238.676161] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1240.360603] Key type lgssc registered [ 1241.516130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.133927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.171081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.709939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.233221] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1262.055209] LustreError: 43560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.075156] LustreError: 43560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1263.076293] LustreError: 43559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.350397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.010665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.414726] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1278.417460] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.876206] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1289.183697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.750345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.320140] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1297.970952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1299.991121] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1307.110298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.122146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.144874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.131205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.131571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.131903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.131907] Lustre: Skipped 1 previous similar message [ 1312.562800] Lustre: lustre-OST0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 1312.578248] Lustre: Skipped 1 previous similar message [ 1313.020946] Lustre: server umount lustre-OST0000 complete [ 1333.350879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.366197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.371855] Lustre: Skipped 1 previous similar message [ 1336.642157] Lustre: server umount lustre-MDT0000 complete [ 1338.851516] LustreError: 44272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.890413] LustreError: 44272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1340.248239] LustreError: 43541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779127766 with bad export cookie 11320844161488441130 [ 1340.268618] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.270699] LustreError: 43541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1346.556211] Lustre: server umount lustre-MDT0001 complete [ 1357.950776] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1360.755097] Key type lgssc unregistered [ 1361.023176] LNet: 46912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.030729] LNetError: 46912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.087985] LNet: Removed LNI 192.168.206.143@tcp [ 1362.863690] Key type .llcrypt unregistered [ 1362.868285] Key type ._llcrypt unregistered [ 1386.213497] Key type ._llcrypt registered [ 1386.215882] Key type .llcrypt registered [ 1386.333146] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1387.377731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.411241] alg: No test for adler32 (adler32-zlib) [ 1388.562868] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1388.806399] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1390.504179] Key type lgssc registered [ 1391.835653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1400.425568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1400.437631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.950397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.486385] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1412.067111] LustreError: 48367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.106566] LustreError: 48367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1415.147795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.432288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.346211] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1424.620729] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1428.218542] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1434.705828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1435.122761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.900937] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1446.390825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1446.571237] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1452.703754] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1455.951586] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1467.921175] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-1814h11m31s [ 1469.245987] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1471.972569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.975595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1471.981263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1471.989923] Lustre: Skipped 1 previous similar message [ 1473.059554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.064495] Lustre: Skipped 4 previous similar messages [ 1474.904271] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 1479.148274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.156317] Lustre: Skipped 3 previous similar messages [ 1485.280202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.286588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.294915] Lustre: Skipped 3 previous similar messages [ 1485.441940] Lustre: server umount lustre-MDT0000 complete [ 1487.165981] LustreError: 48362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.098742] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1491.432289] LustreError: 48367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.452275] LustreError: 48367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1495.887991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.035212] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1496.273521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.367516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1497.341717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1500.701691] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1504.234219] Lustre: DEBUG MARKER: Wait for df (36449) ... [ 1505.812308] Lustre: DEBUG MARKER: done [ 1511.907534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.910058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.924516] LustreError: Skipped 1 previous similar message [ 1511.948932] Lustre: Skipped 1 previous similar message [ 1511.951874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.577660] Lustre: server umount lustre-OST0000 complete [ 1523.170825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.188519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.194776] Lustre: Skipped 1 previous similar message [ 1527.602305] Lustre: server umount lustre-MDT0000 complete [ 1530.340871] LustreError: 48368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.372691] LustreError: 48368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1531.302687] LustreError: 48349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779127957 with bad export cookie 17367288417274305105 [ 1531.304974] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.319929] LustreError: 48349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1531.686567] Lustre: server umount lustre-MDT0001 complete [ 1543.269948] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1546.162420] Key type lgssc unregistered [ 1546.445756] LNet: 52919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.458896] LNetError: 52919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.480500] LNet: Removed LNI 192.168.206.143@tcp [ 1547.316692] Key type .llcrypt unregistered [ 1547.321653] Key type ._llcrypt unregistered [ 1585.930570] Key type ._llcrypt registered [ 1585.933159] Key type .llcrypt registered [ 1586.060385] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1587.014967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.096062] alg: No test for adler32 (adler32-zlib) [ 1588.173230] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1588.434564] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1590.088210] Key type lgssc registered [ 1591.729865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1600.389293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1600.421394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.712896] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1601.733347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1601.915991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.570611] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1617.291728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.400831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1617.456906] Lustre: 55163:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1617.773919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.506875] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1629.555927] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.001300] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1641.417757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.742476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1641.973247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.855781] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1651.829194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1655.436945] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1663.517082] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-1814h14m46s [ 1666.139856] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1675.447832] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1678.779963] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1678.975242] LustreError: 56748:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1699.170301] Lustre: lustre-MDT0000: Client be6fe17f-fa07-4c69-820d-7fbf2d16fd02 (at 192.168.206.43@tcp) reconnecting [ 1719.000188] LustreError: 56748:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1721.978629] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1724.892868] Lustre: DEBUG MARKER: done [ 1733.604852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.626586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.657243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.113264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1738.862845] Lustre: server umount lustre-OST0000 complete [ 1744.355796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.365756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.381434] Lustre: Skipped 2 previous similar messages [ 1744.400696] Lustre: Skipped 3 previous similar messages [ 1745.442525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.454399] Lustre: Skipped 1 previous similar message [ 1749.021810] Lustre: server umount lustre-MDT0000 complete [ 1753.371601] LustreError: 55156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128179 with bad export cookie 4423501296799315990 [ 1753.372738] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.387193] LustreError: 55156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1753.815376] Lustre: server umount lustre-MDT0001 complete [ 1765.969434] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1768.866320] Key type lgssc unregistered [ 1769.260745] LNet: 59141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.274118] LNetError: 59141:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.289920] LNet: Removed LNI 192.168.206.143@tcp [ 1770.258223] Key type .llcrypt unregistered [ 1770.262313] Key type ._llcrypt unregistered [ 1818.893732] Key type ._llcrypt registered [ 1818.901167] Key type .llcrypt registered [ 1819.104530] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1821.076320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.118166] alg: No test for adler32 (adler32-zlib) [ 1822.125982] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1822.438834] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1824.211214] Key type lgssc registered [ 1825.630697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.050616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.084180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.479627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1831.501960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1831.676470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.640364] Lustre: Failing over lustre-MDT0000 [ 1832.990929] Lustre: server umount lustre-MDT0000 complete [ 1838.677248] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:17:42 (1779128262) [ 1844.863036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.275596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.328485] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1857.779330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.859747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.884887] Lustre: 61977:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1858.073661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.674759] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1868.199552] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.048208] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.645765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.872793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1879.056218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.163838] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1885.368109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1891.348544] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1897.108341] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1906.149856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.151896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.170991] Lustre: Skipped 1 previous similar message [ 1906.199491] Lustre: Skipped 2 previous similar messages [ 1913.308921] Lustre: server umount lustre-MDT0000 complete [ 1916.386658] LustreError: 62790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.409259] LustreError: 62790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1916.930266] LustreError: 61969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128342 with bad export cookie 1177890385364746129 [ 1916.950909] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.378970] Lustre: server umount lustre-MDT0001 complete [ 1921.087994] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1927.375217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1937.899354] Lustre: 60867:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128347/real 1779128347] req@00000000566dc839 x1865551181815936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.962333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.998758] Lustre: Skipped 1 previous similar message [ 1940.577066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.048394] LustreError: 65390:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1942.059802] LustreError: 65390:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000a9f97d1a x1865551181816064/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1779128367 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1942.102256] LustreError: 65390:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1942.130158] LustreError: 60864:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000023725575 x1865551181816704/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1942.786684] LustreError: 63596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.917309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.033054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1943.841036] Lustre: 60865:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128352/real 1779128352] req@0000000098923118 x1865551181816192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.919231] LustreError: 63597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.984678] LustreError: 63596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.985593] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1948.000186] Lustre: 60866:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128357/real 1779128357] req@0000000076187bd1 x1865551181816320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1948.160605] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1952.160585] Lustre: 60867:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128362/real 1779128362] req@00000000ed66432b x1865551181816448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1953.250384] LustreError: 65404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.276833] LustreError: 65404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1958.019486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.431608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.073424] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1963.512802] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1970.178225] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1974.604678] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1983.970465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.976679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.980153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.023221] Lustre: Skipped 2 previous similar messages [ 1988.964861] Lustre: server umount lustre-MDT0000 complete [ 1989.090288] LustreError: 66120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.100215] LustreError: 66120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1993.495104] LustreError: 64544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128419 with bad export cookie 1177890385364748404 [ 1993.504916] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1993.514348] LustreError: 64544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1994.148068] Lustre: server umount lustre-MDT0001 complete [ 2002.141759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2010.592315] Lustre: 60868:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128420/real 1779128420] req@00000000aea347fc x1865551181842304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.631668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.566573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.101314] LustreError: 63597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.231392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.354493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2020.323107] Lustre: 60866:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779128429/real 1779128429] req@00000000a5f4f6c3 x1865551181842816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779128445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2020.364714] Lustre: 60866:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2024.271154] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2034.614288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.403033] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2040.798517] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2047.753963] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2052.050782] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2060.260082] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.267514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.267735] Lustre: Skipped 1 previous similar message [ 2060.294596] Lustre: Skipped 4 previous similar messages [ 2065.377539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.389081] Lustre: Skipped 2 previous similar messages [ 2065.868847] Lustre: server umount lustre-MDT0000 complete [ 2070.498718] LustreError: 67860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.528854] LustreError: 67860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2070.923226] LustreError: 61968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128496 with bad export cookie 1177890385364749783 [ 2070.934242] LustreError: 61968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2070.936366] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.296713] Lustre: server umount lustre-MDT0001 complete [ 2079.463279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2082.992544] Lustre: DEBUG MARKER: files compared the same [ 2095.248285] Lustre: server umount lustre-OST0000 complete [ 2114.169369] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 2117.323602] Key type lgssc unregistered [ 2117.653580] LNet: 70896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.659087] LNetError: 70896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.687798] LNet: Removed LNI 192.168.206.143@tcp [ 2118.584513] Key type .llcrypt unregistered [ 2118.590576] Key type ._llcrypt unregistered [ 2143.996044] Key type ._llcrypt registered [ 2143.999397] Key type .llcrypt registered [ 2144.127243] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2145.719200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.761315] alg: No test for adler32 (adler32-zlib) [ 2146.941460] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2147.185888] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 2148.872202] Key type lgssc registered [ 2150.261890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.412854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2159.441559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.216692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.836326] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug +malloc all [ 2170.341777] LustreError: 72349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.358568] LustreError: 72349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2175.476869] LustreError: 72368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.877427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.111259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2180.883216] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug +malloc all [ 2187.991537] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2191.754925] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.261681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.669109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.804121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2207.502440] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug +malloc all [ 2213.904744] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2226.657610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2226.659766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.663429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.665791] LustreError: Skipped 1 previous similar message [ 2226.701600] Lustre: Skipped 1 previous similar message [ 2231.777784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.783738] Lustre: Skipped 1 previous similar message [ 2236.386563] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2236.586042] Lustre: server umount lustre-OST0000 complete [ 2242.017211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.018517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.034971] Lustre: Skipped 1 previous similar message [ 2242.057906] Lustre: Skipped 2 previous similar messages [ 2247.142749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.159361] Lustre: Skipped 1 previous similar message [ 2252.259964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.264871] Lustre: Skipped 1 previous similar message [ 2255.328180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2255.594632] Lustre: server umount lustre-MDT0000 complete [ 2256.353029] LustreError: 72346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.368983] LustreError: 72346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2260.210153] LustreError: 72330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128686 with bad export cookie 3742254442719455145 [ 2260.229377] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.230605] LustreError: 72330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2260.634530] Lustre: server umount lustre-MDT0001 complete [ 2273.209347] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 2276.823810] Key type lgssc unregistered [ 2277.257455] LNet: 75527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.280401] LNetError: 75527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.319533] LNet: Removed LNI 192.168.206.143@tcp [ 2278.434222] Key type .llcrypt unregistered [ 2278.440852] Key type ._llcrypt unregistered [ 2305.724031] Key type ._llcrypt registered [ 2305.729934] Key type .llcrypt registered [ 2305.883854] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2306.597249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.696550] alg: No test for adler32 (adler32-zlib) [ 2307.854803] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2308.091692] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 2309.771245] Key type lgssc registered [ 2310.820365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2319.067898] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2319.099538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.752532] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2339.872331] LustreError: 76954:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.143@tcp: failed processing log, type 4: rc = -110 [ 2366.624221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.177150] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2379.124854] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2386.895060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.024866] Lustre: *** cfs_fail_loc=706, val=0*** [ 2387.438830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.536947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2392.183640] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2392.551667] LustreError: 76980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.568461] LustreError: 76980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2397.671103] LustreError: 76979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.585312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.119789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.849397] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2411.872132] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2415.962060] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2421.736697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.744148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.781785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.758339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.772817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.790031] Lustre: Skipped 1 previous similar message [ 2424.726280] Lustre: server umount lustre-OST0000 complete [ 2431.970203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2431.985664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.018316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.612828] Lustre: server umount lustre-MDT0000 complete [ 2438.120481] LustreError: 77934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.147430] LustreError: 77934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2440.379025] LustreError: 76961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779128866 with bad export cookie 17679481391124470417 [ 2440.389192] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.397198] LustreError: 76961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2441.060622] Lustre: server umount lustre-MDT0001 complete [ 2452.617585] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 2455.323752] Key type lgssc unregistered [ 2455.624993] LNet: 80214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.633293] LNetError: 80214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.649634] LNet: Removed LNI 192.168.206.143@tcp [ 2456.429732] Key type .llcrypt unregistered [ 2456.432240] Key type ._llcrypt unregistered [ 2483.013757] Key type ._llcrypt registered [ 2483.016329] Key type .llcrypt registered [ 2483.142778] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2484.230438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2484.277075] alg: No test for adler32 (adler32-zlib) [ 2485.329114] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2485.613063] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 2487.264204] Key type lgssc registered [ 2488.770754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2497.241139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2497.275137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.088949] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2510.775668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.332584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2515.388657] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2516.451050] LustreError: 82344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.485539] LustreError: 82344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2521.573496] LustreError: 82347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.599405] LustreError: 82347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2524.122903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.501257] LustreError: 82348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.519228] LustreError: 82348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2524.553095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.765827] LustreError: 82343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.801248] LustreError: 82343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2530.405714] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2534.888214] LustreError: 82347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.898646] LustreError: 82347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2538.786680] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.802058] Lustre: MGS: Client 8ece0b6c-9e1c-41ff-b24a-0a2278f6f088 (at 0@lo) reconnecting [ 2538.831963] LustreError: 82358:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2539.148305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.262758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2543.313952] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2559.467540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2559.481959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.521084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.946233] Lustre: server umount lustre-OST0000 complete [ 2570.213733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.221230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2570.221715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.221719] Lustre: Skipped 1 previous similar message [ 2570.237678] Lustre: Skipped 2 previous similar messages [ 2573.615805] Lustre: server umount lustre-MDT0000 complete [ 2575.334387] LustreError: 82344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.363062] LustreError: 82344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2577.549130] Lustre: server umount lustre-MDT0001 complete [ 2580.920251] Lustre: server umount MGS complete [ 2592.257554] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:30:16 (1779129016) [ 2618.889903] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 2628.504636] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2666.320044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.398642] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2681.279676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.437700] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2681.475978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2681.579310] Lustre: lustre-MDT0001: new disk, initializing [ 2681.706798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.747186] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2681.761357] Lustre: 89436:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2681.771759] LustreError: 89436:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2687.004199] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2697.689141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2697.912249] Lustre: lustre-OST0000: new disk, initializing [ 2697.916275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2697.925627] Lustre: 90186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2698.031325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.368660] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2704.417084] Lustre: 90464:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000087dc23a1 x1865551877020800/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779129146 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2711.804594] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2718.905984] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.928204] Lustre: MGS: Client c0cdb933-34f2-43bb-96cd-f113799002c6 (at 0@lo) reconnecting [ 2719.019574] Lustre: 88723:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2719.039815] LustreError: 89422:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2719.051994] LustreError: 89422:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2719.085942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2719.205978] Lustre: lustre-MDT0000: new disk, initializing [ 2719.410825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.438106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2719.466795] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2720.492435] Lustre: 91142:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008ef9ef51 x1865551877027584/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1779129162 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2724.492607] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2725.673902] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2729.646542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2729.664569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2729.734740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2755.556797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.562717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.583431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.587966] Lustre: Skipped 1 previous similar message [ 2757.604535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.616813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.635198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.662369] Lustre: server umount lustre-OST0000 complete [ 2768.354798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.363208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2768.371484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.387828] Lustre: Skipped 3 previous similar messages [ 2770.309069] Lustre: server umount lustre-MDT0000 complete [ 2773.474879] LustreError: 89408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.496141] LustreError: 89408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2780.598497] Lustre: server umount lustre-MDT0001 complete [ 2784.381544] Lustre: server umount MGS complete [ 2795.949275] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:33:39 (1779129219) [ 2814.585987] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 2817.288565] Key type lgssc unregistered [ 2817.568400] LNet: 93608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.572880] LNetError: 93608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.597306] LNet: Removed LNI 192.168.206.143@tcp [ 2818.622967] Key type .llcrypt unregistered [ 2818.628920] Key type ._llcrypt unregistered [ 2836.908379] Key type ._llcrypt registered [ 2836.911735] Key type .llcrypt registered [ 2837.031908] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2838.149413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2838.349697] alg: No test for adler32 (adler32-zlib) [ 2839.482502] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2839.789405] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 2841.526201] Key type lgssc registered [ 2842.780095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.835416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.862208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.532314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.969277] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2863.076325] LustreError: 94866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.094868] LustreError: 94866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2866.901639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.301042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.208253] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2877.962719] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2881.234906] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2887.378641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.627310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.003382] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2897.768188] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2897.911032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2908.128979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.141450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.141623] LustreError: Skipped 1 previous similar message [ 2908.154422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.176684] Lustre: Skipped 1 previous similar message [ 2908.204995] Lustre: Skipped 1 previous similar message [ 2911.935356] Lustre: server umount lustre-OST0000 complete [ 2918.370334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.376245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.379261] Lustre: Skipped 1 previous similar message [ 2918.380489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.412059] Lustre: Skipped 1 previous similar message [ 2922.637637] Lustre: server umount lustre-MDT0000 complete [ 2923.494853] LustreError: 95817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.505885] LustreError: 95817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2926.810514] LustreError: 94846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129352 with bad export cookie 554288474896800621 [ 2926.826910] LustreError: 94846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2926.828286] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.140322] Lustre: server umount lustre-MDT0001 complete [ 2939.407192] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 2942.245468] Key type lgssc unregistered [ 2942.594302] LNet: 98051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.599815] LNetError: 98051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.628587] LNet: Removed LNI 192.168.206.143@tcp [ 2943.523840] Key type .llcrypt unregistered [ 2943.526046] Key type ._llcrypt unregistered [ 2959.000033] Key type ._llcrypt registered [ 2959.007187] Key type .llcrypt registered [ 2959.217807] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2960.843271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.877565] alg: No test for adler32 (adler32-zlib) [ 2962.071925] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2962.346180] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 2964.080160] Key type lgssc registered [ 2965.396878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2972.006277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2972.019748] LustreError: 99154:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2972.167846] LustreError: 99169:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2972.174964] LustreError: 99154:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4860 [ 2972.180406] LustreError: 99154:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2972.189968] LustreError: 99154:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2972.192583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.794520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.180965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.876399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.691902] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2992.543217] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing lsmod [ 2995.267812] LustreError: 100333:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2995.307676] LustreError: 100340:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2995.314919] LustreError: 100333:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2995.323250] LustreError: 100333:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2995.333678] LustreError: 100333:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2995.363309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.780383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.243361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2998.257543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.271313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.805604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.824658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.825589] Lustre: Skipped 1 previous similar message [ 3000.850233] Lustre: Skipped 2 previous similar messages [ 3004.553272] Lustre: server umount lustre-MDT0000 complete [ 3005.922614] LustreError: 99207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.942195] LustreError: 99207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3007.864133] LustreError: 99186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129433 with bad export cookie 16412614472606935295 [ 3007.875617] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.878530] LustreError: 99186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3008.109925] Lustre: server umount lustre-MDT0001 complete [ 3022.293419] Lustre: server umount lustre-OST0000 complete [ 3031.339700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.812311] LustreError: 101404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.873662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.770333] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3037.154667] LustreError: 101405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.258492] LustreError: 101404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.894952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.430125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3050.854798] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3056.797947] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3061.225988] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3068.765975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3069.229621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.633563] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3076.409637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3081.301101] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3090.014263] Lustre: server umount lustre-OST0000 complete [ 3097.066215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.083741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.085855] Lustre: Skipped 1 previous similar message [ 3097.098255] Lustre: Skipped 1 previous similar message [ 3102.178877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.182517] Lustre: Skipped 1 previous similar message [ 3103.254315] Lustre: server umount lustre-MDT0000 complete [ 3106.912227] LustreError: 101387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129532 with bad export cookie 16412614472606936002 [ 3106.917107] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.929760] LustreError: 101387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3116.620285] Lustre: server umount lustre-MDT0001 complete [ 3128.795345] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3131.840645] Key type lgssc unregistered [ 3132.101480] LNet: 104583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3132.111899] LNetError: 104583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3132.135775] LNet: Removed LNI 192.168.206.143@tcp [ 3133.243797] Key type .llcrypt unregistered [ 3133.250035] Key type ._llcrypt unregistered [ 3162.185937] Key type ._llcrypt registered [ 3162.189160] Key type .llcrypt registered [ 3162.322801] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3163.371416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3163.396886] alg: No test for adler32 (adler32-zlib) [ 3164.611623] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 3164.923438] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3166.664529] Key type lgssc registered [ 3167.825701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3175.541806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3175.582442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.028990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.875656] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3186.150598] LustreError: 106038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.170746] LustreError: 106038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3189.297752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.592892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.897559] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3199.889910] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3204.206734] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3210.765194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3211.170652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.933522] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3221.431031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3222.674403] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.429960] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3231.057212] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3236.835167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.836411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3236.841020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3236.847163] Lustre: Skipped 1 previous similar message [ 3241.115383] Lustre: server umount lustre-OST0000 complete [ 3246.051512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3246.059770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.077080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.087175] Lustre: Skipped 1 previous similar message [ 3247.074935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.089629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.094365] Lustre: Skipped 1 previous similar message [ 3250.690658] Lustre: server umount lustre-MDT0000 complete [ 3252.195242] LustreError: 106037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.217061] LustreError: 106037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3253.818802] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129679 with bad export cookie 6840814858639937618 [ 3253.825682] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.834561] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3254.131474] Lustre: server umount lustre-MDT0001 complete [ 3264.123444] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3267.045139] Key type lgssc unregistered [ 3267.367567] LNet: 109317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3267.381593] LNetError: 109317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3268.456607] LNet: Removed LNI 192.168.206.143@tcp [ 3269.365692] Key type .llcrypt unregistered [ 3269.370321] Key type ._llcrypt unregistered [ 3288.311343] Key type ._llcrypt registered [ 3288.314760] Key type .llcrypt registered [ 3288.420761] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3289.565300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3289.577909] alg: No test for adler32 (adler32-zlib) [ 3290.618451] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 3290.877337] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3292.608623] Key type lgssc registered [ 3294.009695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3302.327337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3302.346701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.838860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.098591] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3314.147342] LustreError: 110577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.176314] LustreError: 110577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3317.230769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.623629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.465511] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3328.774977] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3332.472550] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3338.924382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.194180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.377971] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3349.438000] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3351.556681] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3359.716375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.717317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.741647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.750940] Lustre: Skipped 1 previous similar message [ 3364.839602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.849568] Lustre: Skipped 1 previous similar message [ 3365.347371] Lustre: server umount lustre-OST0000 complete [ 3368.934061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.942838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.959817] Lustre: Skipped 1 previous similar message [ 3368.968901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.957292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.808499] Lustre: server umount lustre-MDT0000 complete [ 3375.078947] LustreError: 110576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.097110] LustreError: 110576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3378.374907] LustreError: 110557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129804 with bad export cookie 13286755184303614472 [ 3378.379063] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.390273] LustreError: 110557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3378.899849] Lustre: server umount lustre-MDT0001 complete [ 3390.139562] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3393.125974] Key type lgssc unregistered [ 3393.451646] LNet: 113751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3393.459535] LNetError: 113751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3393.473685] LNet: Removed LNI 192.168.206.143@tcp [ 3394.170578] Key type .llcrypt unregistered [ 3394.174984] Key type ._llcrypt unregistered [ 3416.420289] Key type ._llcrypt registered [ 3416.422526] Key type .llcrypt registered [ 3416.497806] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3417.367593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3417.470730] alg: No test for adler32 (adler32-zlib) [ 3418.571214] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 3418.728039] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3420.344254] Key type lgssc registered [ 3421.184189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3428.406870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3428.428550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.035242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.180274] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3439.606024] LustreError: 115203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.625368] LustreError: 115203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3442.300114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.653817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.723657] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3452.409681] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3455.860389] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3462.587387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.852423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.933518] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3474.167249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3474.779558] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.863257] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3485.116937] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3485.120517] Lustre: Skipped 2 previous similar messages [ 3488.982326] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3488.992486] Lustre: Skipped 2 previous similar messages [ 3491.911412] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3491.959973] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3498.567965] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.43@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3498.584101] Lustre: Skipped 2 previous similar messages [ 3499.954218] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.43@tcp 192.168.0.[2,10]@tcp [ 3510.399623] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.43/24@tcp in log lustre-MDT0000 [ 3510.409985] Lustre: Skipped 2 previous similar messages [ 3517.524910] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.43/24@tcp [ 3517.530809] Lustre: Skipped 1 previous similar message [ 3523.553735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.562872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3523.570551] Lustre: Skipped 1 previous similar message [ 3523.598176] Lustre: Skipped 1 previous similar message [ 3525.659091] Lustre: server umount lustre-OST0000 complete [ 3529.697570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3529.704064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.723182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.795064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.795277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.824689] Lustre: Skipped 1 previous similar message [ 3535.624181] Lustre: server umount lustre-MDT0000 complete [ 3538.735044] LustreError: 115186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779129964 with bad export cookie 2494076118411502671 [ 3538.743735] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.754390] LustreError: 115186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3539.046851] Lustre: server umount lustre-MDT0001 complete [ 3548.752724] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3551.145902] Key type lgssc unregistered [ 3551.410624] LNet: 119732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3551.422949] LNetError: 119732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3551.433794] LNet: Removed LNI 192.168.206.143@tcp [ 3552.021667] Key type .llcrypt unregistered [ 3552.024783] Key type ._llcrypt unregistered [ 3579.282944] Key type ._llcrypt registered [ 3579.285427] Key type .llcrypt registered [ 3579.399263] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3580.331329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.464555] alg: No test for adler32 (adler32-zlib) [ 3581.548627] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 3581.747509] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3583.416181] Key type lgssc registered [ 3584.335284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3591.780975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3591.828690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.366343] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3593.416785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.424614] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.43/24@tcp [ 3599.040237] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3603.937453] LustreError: 121329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.987435] LustreError: 121329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3609.062153] LustreError: 121329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.546592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.799548] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3609.817546] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.43/24@tcp [ 3609.877975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.061251] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3622.740230] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3627.639834] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3635.346385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.702453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.784138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3642.217883] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3649.414600] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3663.859622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3663.864194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.876213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.397165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.397842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3665.398901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.403289] Lustre: Skipped 1 previous similar message [ 3669.854535] Lustre: server umount lustre-OST0000 complete [ 3675.617822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.620573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.644447] Lustre: Skipped 1 previous similar message [ 3675.656475] Lustre: Skipped 1 previous similar message [ 3679.905564] Lustre: server umount lustre-MDT0000 complete [ 3680.748094] LustreError: 122013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.772141] LustreError: 122013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3683.603401] LustreError: 121309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779130109 with bad export cookie 14923297408288452394 [ 3683.608061] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.619420] LustreError: 121309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3684.032855] Lustre: server umount lustre-MDT0001 complete [ 3696.334424] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3699.228935] Key type lgssc unregistered [ 3699.564070] LNet: 124563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.575421] LNetError: 124563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3700.647199] LNet: Removed LNI 192.168.206.143@tcp [ 3701.482182] Key type .llcrypt unregistered [ 3701.485308] Key type ._llcrypt unregistered [ 3730.332292] Key type ._llcrypt registered [ 3730.334780] Key type .llcrypt registered [ 3730.476237] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 3747.472788] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3748.710979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.752361] alg: No test for adler32 (adler32-zlib) [ 3749.841729] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 3750.108375] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3751.896370] Key type lgssc registered [ 3753.269778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3788.632175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3788.665695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.878713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3789.902421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3789.974767] Lustre: lustre-MDT0000: new disk, initializing [ 3790.078671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.111605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3793.745717] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3804.540426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.699150] Lustre: 127990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.825411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3804.831909] Lustre: Skipped 1 previous similar message [ 3804.984492] Lustre: lustre-MDT0001: new disk, initializing [ 3805.097470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3805.128469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3805.139767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3808.954139] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3816.990191] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3821.455505] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3827.901617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.160310] Lustre: lustre-OST0000: new disk, initializing [ 3828.163768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3828.170323] Lustre: 129731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3828.245609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.422523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.432662] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3834.438916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.569941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3842.327908] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3846.804703] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3847.016890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3851.526821] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3851.768700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3855.340093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.360594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.362443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.378176] Lustre: Skipped 1 previous similar message [ 3860.031899] Lustre: server umount lustre-OST0000 complete [ 3865.572245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.589976] Lustre: Skipped 1 previous similar message [ 3865.592768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.601121] Lustre: Skipped 1 previous similar message [ 3870.692542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3870.699509] Lustre: Skipped 1 previous similar message [ 3875.809192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3875.822339] Lustre: Skipped 1 previous similar message [ 3878.369286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3878.637164] Lustre: server umount lustre-MDT0000 complete [ 3882.713900] LustreError: 127981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779130308 with bad export cookie 17848737035876335382 [ 3882.717869] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.731310] LustreError: 127981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3883.209659] Lustre: server umount lustre-MDT0001 complete [ 3891.742168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.284412] LustreError: 131802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.304293] LustreError: 131802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3892.366669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.661599] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3897.825172] LustreError: 131803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.947465] LustreError: 131802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.433154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.914946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3910.433705] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3916.394140] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3920.530726] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3928.089435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3928.508030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.604464] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3941.612780] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3946.899174] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3947.075548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3951.287485] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3951.522931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3960.189830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.427974] Lustre: lustre-OST0001: new disk, initializing [ 3960.436834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3960.450494] Lustre: 134798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3960.588086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.009131] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3968.127579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3968.145911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3968.262064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3975.909605] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3976.165611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3980.633204] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3980.913369] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3985.526513] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c7fc5833800.ost_server_uuid 50 [ 3987.678148] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c7fc5833800.ost_server_uuid in FULL state after 0 sec [ 3999.206756] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.214357] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.231282] Lustre: Skipped 1 previous similar message [ 3999.232917] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4003.768689] Lustre: server umount lustre-OST0001 complete [ 4004.326969] LustreError: 133398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.350042] LustreError: 133398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4009.454144] LustreError: 133862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.458969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.472720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.472730] Lustre: Skipped 1 previous similar message [ 4009.473494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.473498] Lustre: Skipped 1 previous similar message [ 4009.478215] LustreError: 133862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4009.500705] LustreError: Skipped 1 previous similar message [ 4013.919431] Lustre: server umount lustre-OST0000 complete [ 4018.658836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4018.676696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.694825] Lustre: Skipped 1 previous similar message [ 4024.372468] Lustre: server umount lustre-MDT0000 complete [ 4024.810845] LustreError: 131802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.842977] LustreError: 131802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4028.021895] LustreError: 131782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779130453 with bad export cookie 17848737035876336264 [ 4028.030110] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.035156] LustreError: 131782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4028.618907] Lustre: server umount lustre-MDT0001 complete [ 4049.505711] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4053.390838] Key type lgssc unregistered [ 4053.880133] LNet: 137636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4053.890613] LNetError: 137636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4054.963931] LNet: Removed LNI 192.168.206.143@tcp [ 4056.029605] Key type .llcrypt unregistered [ 4056.047258] Key type ._llcrypt unregistered [ 4097.667723] Key type ._llcrypt registered [ 4097.670917] Key type .llcrypt registered [ 4097.767862] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 4112.210590] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4113.240079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4113.280426] alg: No test for adler32 (adler32-zlib) [ 4114.331390] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 4114.555959] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4116.248193] Key type lgssc registered [ 4117.358319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4151.976549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4151.990225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.337974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4153.403768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4153.538954] Lustre: lustre-MDT0000: new disk, initializing [ 4153.697845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.730784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4158.999341] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4178.102184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.312065] Lustre: 141813:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4178.446589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4178.458650] Lustre: Skipped 1 previous similar message [ 4178.695940] Lustre: lustre-MDT0001: new disk, initializing [ 4178.996576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4179.086557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4179.121467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4184.604859] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4195.529211] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4199.548363] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4206.956356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.289891] Lustre: lustre-OST0000: new disk, initializing [ 4207.296284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4207.306719] Lustre: 143558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4207.414048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.301727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4213.308639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4213.416965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4214.070303] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4221.097408] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4226.006131] Lustre: Failing over lustre-OST0000 [ 4226.201929] Lustre: server umount lustre-OST0000 complete [ 4243.425109] Lustre: 140006:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779130653/real 1779130653] req@0000000008d04317 x1865553585144960/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1779130669 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4243.454776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.512173] Lustre: 140006:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779130654/real 1779130654] req@000000004661a3dd x1865553585145984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779130670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4244.513582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.541071] Lustre: 140006:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4245.020603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4245.520785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.537345] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4250.418540] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4250.593677] Lustre: 140004:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779130660/real 1779130660] req@000000006eaeec9f x1865553585147136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779130676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4251.576414] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4254.796055] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4254.798168] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4255.015090] Lustre: Failing over lustre-MDT0000 [ 4255.306552] Lustre: server umount lustre-MDT0000 complete [ 4255.583538] LustreError: 143006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.604513] LustreError: 143006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4255.719354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4255.732869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4255.842580] Lustre: 140004:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779130665/real 1779130665] req@00000000cb4450f0 x1865553585148288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779130681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4255.886100] Lustre: 140004:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4258.786061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.797685] Lustre: Skipped 1 previous similar message [ 4258.800935] LustreError: 141820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.819032] LustreError: 141820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4259.872159] Lustre: 140005:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779130669/real 1779130669] req@00000000db0e70f5 x1865553585149440/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779130685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4260.663520] LustreError: 141822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.908136] LustreError: 141826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.926680] LustreError: 141826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4269.038378] LustreError: 141825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.071121] LustreError: 141825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4274.379231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.591336] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4274.870570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.947869] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4277.041552] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4279.410553] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4280.317971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4280.323563] Lustre: Skipped 1 previous similar message [ 4280.355817] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4280.398979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4290.530648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.550144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.865184] Lustre: server umount lustre-OST0000 complete [ 4300.769320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4300.770521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.781832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4300.803321] Lustre: Skipped 2 previous similar messages [ 4300.826017] Lustre: Skipped 2 previous similar messages [ 4304.112176] Lustre: server umount lustre-MDT0000 complete [ 4305.897145] LustreError: 141825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.921342] LustreError: 141825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4307.272479] LustreError: 141805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779130733 with bad export cookie 8955303938778365702 [ 4307.277020] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.287803] LustreError: 141805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4307.545583] Lustre: server umount lustre-MDT0001 complete [ 4317.937658] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4320.139271] Key type lgssc unregistered [ 4320.393539] LNet: 147056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4320.405111] LNetError: 147056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4320.423462] LNet: Removed LNI 192.168.206.143@tcp [ 4321.168147] Key type .llcrypt unregistered [ 4321.170497] Key type ._llcrypt unregistered [ 4360.361652] Key type ._llcrypt registered [ 4360.373778] Key type .llcrypt registered [ 4360.622387] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4361.506316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4361.761317] alg: No test for adler32 (adler32-zlib) [ 4364.010287] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 4364.441748] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4366.360463] Key type lgssc registered [ 4368.387260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4380.126818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4380.169529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.738388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.745537] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4391.910336] LustreError: 148614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.942567] LustreError: 148614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4396.246214] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4397.025938] LustreError: 148613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.547858] LustreError: 148642:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4400.564130] LustreError: 148642:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4400.578675] LustreError: 148642:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4401.201251] Lustre: server umount lustre-MDT0000 complete [ 4420.951228] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 4431.423097] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4472.035316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.463927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4472.536877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4472.608712] Lustre: lustre-MDT0000: new disk, initializing [ 4472.672866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.686194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4477.384128] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4489.715530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.860086] Lustre: 151821:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4489.926831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4489.932646] Lustre: Skipped 1 previous similar message [ 4490.041222] Lustre: lustre-MDT0001: new disk, initializing [ 4490.142934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4490.185527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4490.196679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4495.435691] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4503.987450] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4508.148753] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4516.016908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.252388] Lustre: lustre-OST0000: new disk, initializing [ 4516.255653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4516.260455] Lustre: 153561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.363567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.718598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4523.733325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4523.871637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4524.003774] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4533.730851] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4540.069739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4542.481359] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:02:45 (1779130965) [ 4833.248639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4833.253345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.275176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4836.324869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.327736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4836.344776] Lustre: Skipped 1 previous similar message [ 4836.375351] Lustre: Skipped 2 previous similar messages [ 4840.212987] Lustre: server umount lustre-MDT0000 complete [ 4841.441753] LustreError: 155211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.488658] LustreError: 155211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4844.966290] LustreError: 151814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779131270 with bad export cookie 1144840245219486962 [ 4844.967802] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4844.973631] LustreError: 151814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4846.563177] LustreError: 151828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.571354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.637070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4849.576317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4854.689320] LustreError: 152828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.690647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4854.721397] LustreError: 152828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4859.360129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4859.792661] Lustre: server umount lustre-MDT0001 complete [ 4869.843943] Lustre: server umount lustre-OST0000 complete [ 4880.801443] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 4892.425344] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4937.268021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.563163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4937.599094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4937.682219] Lustre: lustre-MDT0000: new disk, initializing [ 4937.808100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.830095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4942.634900] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4955.568836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.710357] Lustre: 158099:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4955.756904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4955.760617] Lustre: Skipped 1 previous similar message [ 4955.837443] Lustre: lustre-MDT0001: new disk, initializing [ 4955.925825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4955.980958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4955.996208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4961.013780] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4969.514905] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4973.774937] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4980.123510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.358890] Lustre: lustre-OST0000: new disk, initializing [ 4980.363894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4980.373232] Lustre: 159842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4980.459155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.280309] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4987.959208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4987.971815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4988.136470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4994.955562] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4999.278996] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4999.521668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5004.081151] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5004.327943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5008.873192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.873699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.883381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.895707] LustreError: Skipped 1 previous similar message [ 5008.940344] Lustre: Skipped 1 previous similar message [ 5012.667226] Lustre: server umount lustre-OST0000 complete [ 5017.570957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5017.582199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.225133] LustreError: 158111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.249530] LustreError: 158111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5024.818985] Lustre: server umount lustre-MDT0000 complete [ 5028.964528] LustreError: 159841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779131454 with bad export cookie 1144840245219645792 [ 5028.970592] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.977512] LustreError: 159841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5029.396984] Lustre: server umount lustre-MDT0001 complete [ 5038.941076] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:11:02 (1779131462) [ 5055.407866] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 5065.966692] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5104.717921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.962617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5104.994990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5105.085409] Lustre: lustre-MDT0000: new disk, initializing [ 5105.180253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5105.218953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5109.634834] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5121.216838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.426115] Lustre: 164424:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5121.535273] LustreError: 164434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.557438] LustreError: 164434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5121.829531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5127.249741] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5135.743600] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5139.925514] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5146.768456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.149220] Lustre: lustre-OST0000: new disk, initializing [ 5147.151691] Lustre: Skipped 1 previous similar message [ 5147.155479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5147.160037] Lustre: Skipped 2 previous similar messages [ 5147.165551] Lustre: 166173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5154.470264] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5155.429751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5155.438984] Lustre: Skipped 1 previous similar message [ 5155.462967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5155.628919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5162.446406] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5171.169247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.173119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.180409] LustreError: Skipped 1 previous similar message [ 5171.193102] Lustre: Skipped 1 previous similar message [ 5171.194905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.205848] Lustre: Skipped 5 previous similar messages [ 5176.169459] Lustre: server umount lustre-OST0000 complete [ 5186.402826] Lustre: server umount lustre-MDT0000 complete [ 5186.535787] LustreError: 164435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.421284] LustreError: 164415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779131617 with bad export cookie 1144840245219646674 [ 5191.428978] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.442992] LustreError: 164415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5192.236586] Lustre: server umount lustre-MDT0001 complete [ 5202.825402] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:13:46 (1779131626) [ 5223.313645] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 5237.116664] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5277.907592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.169924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5278.204486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5278.284687] Lustre: lustre-MDT0000: new disk, initializing [ 5278.343749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.346553] Lustre: Skipped 2 previous similar messages [ 5278.363966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5282.911465] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5294.269505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.418652] Lustre: 170363:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5294.663462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5299.306273] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5308.234579] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5312.661520] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5319.791782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5320.104074] Lustre: 172114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5326.519691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5326.574217] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5335.162873] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.796469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5346.809300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.836863] Lustre: Skipped 3 previous similar messages [ 5346.851726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5346.854648] Lustre: Skipped 2 previous similar messages [ 5349.724387] Lustre: server umount lustre-OST0000 complete [ 5356.004539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5356.015755] LustreError: Skipped 1 previous similar message [ 5360.018983] Lustre: server umount lustre-MDT0000 complete [ 5362.663336] LustreError: 170370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5362.687638] LustreError: 170370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5365.111568] LustreError: 170354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779131790 with bad export cookie 1144840245219647759 [ 5365.130562] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.597569] Lustre: server umount lustre-MDT0001 complete [ 5378.065264] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 5381.523835] Key type lgssc unregistered [ 5381.868747] LNet: 173843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5381.878347] LNetError: 173843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5381.899291] LNet: Removed LNI 192.168.206.143@tcp [ 5382.803255] Key type .llcrypt unregistered [ 5382.805518] Key type ._llcrypt unregistered [ 5413.040965] Key type ._llcrypt registered [ 5413.043791] Key type .llcrypt registered [ 5413.154631] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5414.161678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5414.317420] alg: No test for adler32 (adler32-zlib) [ 5415.436949] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 5415.724290] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 5417.416266] Key type lgssc registered [ 5418.868193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5428.064917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5428.101743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.705247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.785085] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5439.972047] LustreError: 175295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.999400] LustreError: 175295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5444.369926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.004302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5449.404908] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5456.151084] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5461.114225] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5469.404182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5469.677468] LustreError: 176888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5469.897916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.022537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5478.554302] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5485.158383] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5506.018234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.023601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5506.057872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5506.070432] Lustre: Skipped 1 previous similar message [ 5509.456184] Lustre: server umount lustre-OST0000 complete [ 5516.258351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.267329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.280648] Lustre: Skipped 2 previous similar messages [ 5516.292867] Lustre: Skipped 1 previous similar message [ 5520.516699] Lustre: server umount lustre-MDT0000 complete [ 5521.386982] LustreError: 176005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.406048] LustreError: 176005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5525.039148] LustreError: 175275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779131950 with bad export cookie 13614864899859342651 [ 5525.041138] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.059904] LustreError: 175275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5525.588534] Lustre: server umount lustre-MDT0001 complete [ 5538.639947] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 5541.757248] Key type lgssc unregistered [ 5542.159286] LNet: 178498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5542.169645] LNetError: 178498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5542.192865] LNet: Removed LNI 192.168.206.143@tcp [ 5543.224308] Key type .llcrypt unregistered [ 5543.234805] Key type ._llcrypt unregistered [ 5569.802991] Key type ._llcrypt registered [ 5569.809532] Key type .llcrypt registered [ 5569.947467] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5571.563257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5571.613601] alg: No test for adler32 (adler32-zlib) [ 5572.787963] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 5573.039690] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 5574.744229] Key type lgssc registered [ 5575.860852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5584.175446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5584.247059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.996402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.538341] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5596.133843] LustreError: 179956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.150271] LustreError: 179956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5600.980821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.255566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5601.440534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5606.263512] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5612.367724] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5616.687226] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5623.930806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5624.332419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.336312] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5631.481781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5638.479964] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5645.647885] Lustre: lustre-OST0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 5646.823583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.824099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.831686] LustreError: Skipped 1 previous similar message [ 5646.836445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.859769] Lustre: Skipped 1 previous similar message [ 5650.744250] Lustre: lustre-OST0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 5650.756087] Lustre: Skipped 1 previous similar message [ 5651.696433] Lustre: server umount lustre-OST0000 complete [ 5658.814374] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c7fd0453000.ost_server_uuid 50 [ 5660.947546] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5676.515703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.535653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5676.541826] Lustre: Skipped 1 previous similar message [ 5680.922098] Lustre: server umount lustre-MDT0000 complete [ 5681.640284] LustreError: 179956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.668077] LustreError: 179956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5684.314609] LustreError: 179936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132110 with bad export cookie 11117044188743995653 [ 5684.317759] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5684.324904] LustreError: 179936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5684.666063] Lustre: server umount lustre-MDT0001 complete [ 5692.884240] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:21:56 (1779132116) [ 5698.702418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.014332] LustreError: 183431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.093583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.994387] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5704.167606] LustreError: 183432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.262123] LustreError: 183431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.300721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.714064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.467913] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5722.540922] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5726.255246] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5733.158254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.604938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.944843] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5744.902146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5745.007151] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.851642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5752.066458] Lustre: lustre-OST0001: new disk, initializing [ 5752.070652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5752.076645] Lustre: 185931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5752.177933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5758.366951] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5760.606329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5760.614947] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5760.710250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5766.678554] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5770.722912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.728543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.766078] Lustre: Skipped 1 previous similar message [ 5770.781945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.576977] Lustre: server umount lustre-OST0000 complete [ 5776.375791] LustreError: 185026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.415848] LustreError: 185026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5784.310841] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5784.647204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5790.890139] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5791.266653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5793.074044] LustreError: 185816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.108586] LustreError: 185816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5807.078076] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.078955] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.087180] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5807.087191] Lustre: Skipped 3 previous similar messages [ 5807.102665] Lustre: Skipped 2 previous similar messages [ 5811.407980] Lustre: server umount lustre-OST0001 complete [ 5817.315787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.338381] Lustre: Skipped 1 previous similar message [ 5822.511692] Lustre: server umount lustre-MDT0000 complete [ 5826.428497] LustreError: 183413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132252 with bad export cookie 11117044188743996717 [ 5826.441920] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.456280] LustreError: 183413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5826.863601] Lustre: server umount lustre-MDT0001 complete [ 5851.954065] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:24:35 (1779132275) [ 5859.214366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.477423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5859.547884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5859.817306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.147575] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5872.590123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.700458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5872.761091] Lustre: 189224:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5878.021585] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5885.757432] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5890.535675] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5897.774483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5897.988024] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5904.524808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5904.743691] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5911.661855] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5921.856245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.065687] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5931.664171] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5939.419589] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5945.313057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5945.332822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.355762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.366952] Lustre: Skipped 3 previous similar messages [ 5948.874889] Lustre: server umount lustre-OST0000 complete [ 5950.785207] LustreError: 190850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.805839] LustreError: 190850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5966.312149] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.314299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.342045] Lustre: Skipped 1 previous similar message [ 5966.357989] LustreError: Skipped 1 previous similar message [ 5970.427137] Lustre: server umount lustre-OST0001 complete [ 5975.530296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5975.542776] LustreError: Skipped 1 previous similar message [ 5979.851151] Lustre: server umount lustre-MDT0000 complete [ 5983.211720] LustreError: 189216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132409 with bad export cookie 11117044188743998131 [ 5983.222698] LustreError: 189216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5983.228838] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.521409] Lustre: server umount lustre-MDT0001 complete [ 6005.295855] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:27:09 (1779132429) [ 6021.874478] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 6031.290353] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 6070.085289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.444827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6070.524514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6070.666946] Lustre: lustre-MDT0000: new disk, initializing [ 6070.804293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.809849] Lustre: Skipped 3 previous similar messages [ 6070.821053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6075.430173] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6085.896127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.002992] Lustre: 196843:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6086.043691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6086.046577] Lustre: Skipped 1 previous similar message [ 6086.138599] Lustre: lustre-MDT0001: new disk, initializing [ 6086.286637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6086.305386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6090.773609] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6098.586845] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6102.318927] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6109.646025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6109.963543] Lustre: lustre-OST0000: new disk, initializing [ 6109.967969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6109.985617] Lustre: 198581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6116.217793] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6116.421656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6116.440294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6116.570853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6123.814769] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6127.850770] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6128.052552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6132.523359] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6132.766543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6137.316292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6137.318483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.325268] LustreError: Skipped 1 previous similar message [ 6137.326546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.326561] Lustre: Skipped 7 previous similar messages [ 6137.357964] Lustre: Skipped 4 previous similar messages [ 6140.873940] Lustre: server umount lustre-OST0000 complete [ 6151.098890] Lustre: server umount lustre-MDT0000 complete [ 6152.677151] LustreError: 196852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.699359] LustreError: 196852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6155.066177] LustreError: 196834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132580 with bad export cookie 11117044188743999692 [ 6155.081197] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.083142] LustreError: 196834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6155.431534] Lustre: server umount lustre-MDT0001 complete [ 6164.311586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.448846] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6178.068875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.770297] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6188.716766] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6192.608506] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6199.541457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.982859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.993117] Lustre: Skipped 4 previous similar messages [ 6205.912582] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6210.977395] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6215.340525] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6215.532736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6219.059183] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6219.257797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6224.867043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.867966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6224.888834] Lustre: Skipped 2 previous similar messages [ 6224.908817] LustreError: Skipped 1 previous similar message [ 6227.119674] Lustre: server umount lustre-OST0000 complete [ 6232.412210] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6236.995041] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6265.870164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.617568] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6276.791929] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6284.724078] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6288.917649] Lustre: server umount lustre-OST0000 complete [ 6295.011402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.017259] Lustre: Skipped 6 previous similar messages [ 6298.808671] Lustre: server umount lustre-MDT0000 complete [ 6300.129596] LustreError: 202752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.161115] LustreError: 202752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6302.186028] LustreError: 200634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132728 with bad export cookie 11117044188744000574 [ 6302.188421] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.199309] LustreError: 200634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6302.480941] Lustre: server umount lustre-MDT0001 complete [ 6310.464625] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:32:14 (1779132734) [ 6316.480525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.125902] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6329.572744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.750931] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6341.397988] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6346.181680] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6354.354031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.538181] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6368.429814] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6372.987892] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6373.224857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6377.571729] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6377.838677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6384.680812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.929426] Lustre: lustre-OST0001: new disk, initializing [ 6384.948739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6384.962646] Lustre: 209088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6391.692455] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6391.946338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6391.965609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6392.134774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6400.206355] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6405.976674] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6406.454101] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6412.953475] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6413.300458] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6417.891365] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.892342] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6417.897501] LustreError: Skipped 1 previous similar message [ 6417.930019] Lustre: Skipped 4 previous similar messages [ 6422.233537] Lustre: server umount lustre-OST0001 complete [ 6430.455970] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6430.712223] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6434.940762] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6435.232700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6464.128281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.344047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6464.353508] Lustre: Skipped 5 previous similar messages [ 6469.609692] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6469.622659] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6469.630136] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6469.965178] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6475.715201] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6477.839724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6494.176338] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6494.292086] Lustre: server umount lustre-OST0001 complete [ 6517.092610] Lustre: server umount lustre-OST0000 complete [ 6526.583920] Lustre: server umount lustre-MDT0000 complete [ 6529.895918] LustreError: 209087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779132955 with bad export cookie 11117044188744001519 [ 6529.897876] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.909559] LustreError: 209087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6544.352537] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6544.717769] Lustre: server umount lustre-MDT0001 complete [ 6568.062625] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:36:32 (1779132992) [ 6573.896692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.328456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6574.386644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6579.167667] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6586.933947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.010858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6587.044581] Lustre: 214040:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6590.734370] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6595.254241] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6598.644324] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6604.885838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.054509] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6611.074262] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6616.172925] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6624.304780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.403786] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6631.402299] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6637.575777] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6639.280777] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6643.784265] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6644.037876] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6648.418715] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6648.617610] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6652.261752] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c7fc2e63800.ost_server_uuid 50 [ 6653.783370] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c7fc2e63800.ost_server_uuid in FULL state after 0 sec [ 6654.679505] Lustre: Permanently deactivating lustre-OST0001 [ 6654.695958] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6658.242404] Lustre: Permanently reactivating lustre-OST0001 [ 6665.718966] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6665.731934] LustreError: Skipped 3 previous similar messages [ 6665.733815] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6665.748189] Lustre: Skipped 19 previous similar messages [ 6667.994370] Lustre: server umount lustre-OST0001 complete [ 6670.832302] LustreError: 215659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.857469] LustreError: 215659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6675.426044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.447933] Lustre: Skipped 8 previous similar messages [ 6678.709433] Lustre: server umount lustre-OST0000 complete [ 6689.088752] Lustre: server umount lustre-MDT0000 complete [ 6693.053706] LustreError: 217866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779133118 with bad export cookie 11117044188744002814 [ 6693.058376] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6693.069688] LustreError: 217866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6693.401591] Lustre: server umount lustre-MDT0001 complete [ 6716.643994] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:39:00 (1779133140) [ 6724.754243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.108450] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6725.149387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6725.160307] Lustre: Skipped 5 previous similar messages [ 6729.981251] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6739.763615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.893979] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6739.920570] Lustre: 220150:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6744.337310] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6750.375614] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6754.390937] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6763.118996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.333804] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6763.366504] Lustre: Permanently deactivating lustre-OST0000 [ 6763.380753] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6770.750823] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6771.220904] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6771.227839] LustreError: 222153:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6771.240695] LustreError: 222153:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6777.753579] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6786.405106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.535923] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6794.019650] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6794.187970] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6800.718530] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6802.979678] Lustre: Permanently reactivating lustre-OST0000 [ 6803.006538] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6803.019502] Lustre: Skipped 2 previous similar messages [ 6812.210783] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6812.211203] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000c45a7257, cur 1779133238 deadline 1779133227 last 1779133197 [ 6812.222039] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6812.253712] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6812.262104] Lustre: Skipped 1 previous similar message [ 6812.302053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6824.373620] Lustre: server umount lustre-OST0001 complete [ 6834.716418] Lustre: server umount lustre-OST0000 complete [ 6844.444273] Lustre: server umount lustre-MDT0000 complete [ 6848.627419] LustreError: 222689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779133274 with bad export cookie 11117044188744004452 [ 6848.630604] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.645827] LustreError: 222689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6849.087389] Lustre: server umount lustre-MDT0001 complete [ 6860.928737] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 6864.151870] Key type lgssc unregistered [ 6864.427936] LNet: 224563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6864.439225] LNetError: 224563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6864.464836] LNet: Removed LNI 192.168.206.143@tcp [ 6865.246703] Key type .llcrypt unregistered [ 6865.249070] Key type ._llcrypt unregistered [ 6890.467335] Key type ._llcrypt registered [ 6890.470934] Key type .llcrypt registered [ 6890.643259] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 6892.743909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6892.775370] alg: No test for adler32 (adler32-zlib) [ 6893.948927] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 6894.227799] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 6895.976187] Key type lgssc registered [ 6897.515402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6912.228268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6912.292456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.198184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.006860] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6923.755431] LustreError: 226068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.789155] LustreError: 226068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6928.871214] LustreError: 226069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6930.421798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.585277] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6930.612570] Lustre: 226057:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.624217] Lustre: Permanently deactivating lustre-MDT0001 [ 6930.636709] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6930.944287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6936.111869] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6938.663439] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6942.468291] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6946.773930] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6954.632105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.058784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.079252] LustreError: 227683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.144503] LustreError: 227684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.171797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6959.175471] LustreError: 227684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6961.953815] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6964.203312] LustreError: 227683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.226289] LustreError: 227683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6968.440463] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6974.439843] LustreError: 227685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.471190] LustreError: 227685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6976.161202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.569519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6981.649232] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6981.649987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6983.831761] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 6989.999803] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6994.322069] Lustre: Permanently reactivating lustre-MDT0001 [ 6994.349283] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6994.363612] Lustre: Skipped 2 previous similar messages [ 6997.125351] Lustre: lustre-MDT0001: Client e230b48d-1640-4106-bc06-897bc25a6bc7 (at 192.168.206.43@tcp) reconnecting [ 7000.670292] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.679077] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7000.685798] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7000.695389] LustreError: 229232:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7000.702856] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7000.709845] LustreError: 229232:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7000.709978] LustreError: 229232:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.143@tcp: cfg command failed: rc = -108 [ 7000.717311] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7000.717311] [ 7000.718656] LustreError: 226060:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7001.724426] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7004.654935] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7013.128717] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7013.441660] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7017.908206] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7018.321991] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7022.287865] Lustre: Permanently deactivating lustre-MDT0001 [ 7022.315738] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7022.330530] Lustre: Skipped 2 previous similar messages [ 7031.857477] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7042.032633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.044570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7042.046472] Lustre: Skipped 1 previous similar message [ 7042.057301] Lustre: Skipped 3 previous similar messages [ 7044.328708] Lustre: server umount lustre-MDT0000 complete [ 7047.151327] LustreError: 226068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.177229] LustreError: 226068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7048.481115] LustreError: 229165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779133474 with bad export cookie 15141734291256881942 [ 7048.490427] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.505344] LustreError: 229165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7049.104284] Lustre: server umount lustre-MDT0001 complete [ 7055.764817] Lustre: server umount lustre-OST0000 complete [ 7068.014883] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:44:51 (1779133491) [ 7080.480560] Lustre: server umount lustre-OST0001 complete [ 7088.362094] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 7101.759248] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7138.365095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.642272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7138.650993] Lustre: Skipped 2 previous similar messages [ 7138.682534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7138.757513] Lustre: lustre-MDT0000: new disk, initializing [ 7138.871099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.902725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7143.562635] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7154.955818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.074152] Lustre: 233773:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.133331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7155.137735] Lustre: Skipped 1 previous similar message [ 7155.252609] Lustre: lustre-MDT0001: new disk, initializing [ 7155.376613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7155.444583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7155.456701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7160.154196] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7168.429869] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7173.237702] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7181.734035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.000834] Lustre: lustre-OST0000: new disk, initializing [ 7182.006947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7182.015515] Lustre: 235514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7182.092916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.508287] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7191.617587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7191.629070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7191.727706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7197.006641] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.466791] LustreError: 235958:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7208.482531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7208.749543] Lustre: lustre-OST0001: new disk, initializing [ 7208.764728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7208.782585] Lustre: 236515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7209.482153] LustreError: 235958:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7209.517349] LustreError: 233780:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7215.229140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7215.247152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7215.364684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7215.657095] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7219.552556] LustreError: 233780:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7223.177413] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7224.967228] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7229.792595] Lustre: lustre-OST0001: Not available for connect from 192.168.206.43@tcp (stopping) [ 7230.950793] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7230.953994] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.966973] Lustre: Skipped 2 previous similar messages [ 7230.971965] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7230.976276] Lustre: Skipped 1 previous similar message [ 7233.254076] Lustre: server umount lustre-OST0001 complete [ 7234.873545] LustreError: 235495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.430209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.450090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7251.467370] Lustre: Skipped 2 previous similar messages [ 7256.549984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7256.554909] Lustre: Skipped 2 previous similar messages [ 7265.248691] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7265.561631] Lustre: server umount lustre-OST0000 complete [ 7271.910268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.920248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.930219] Lustre: Skipped 1 previous similar message [ 7271.954675] Lustre: Skipped 3 previous similar messages [ 7275.858292] Lustre: server umount lustre-MDT0000 complete [ 7279.698481] LustreError: 233765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779133705 with bad export cookie 15141734291256884539 [ 7279.710676] LustreError: 233765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7279.712275] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7286.152940] Lustre: server umount lustre-MDT0001 complete [ 7297.189597] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7300.016297] Key type lgssc unregistered [ 7300.357761] LNet: 238419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7300.376417] LNetError: 238419:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.415487] LNet: Removed LNI 192.168.206.143@tcp [ 7302.420834] Key type .llcrypt unregistered [ 7302.423248] Key type ._llcrypt unregistered [ 7343.905050] Key type ._llcrypt registered [ 7343.907137] Key type .llcrypt registered [ 7344.013551] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7345.093348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7345.138409] alg: No test for adler32 (adler32-zlib) [ 7346.149922] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 7346.368179] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7348.056151] Key type lgssc registered [ 7349.239093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7357.443675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7357.484496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.888399] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7358.932741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7359.085629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.908478] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7371.412826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.544938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7371.595569] Lustre: 240665:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7371.875803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7376.402473] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7382.567831] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7386.865162] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7394.279319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7394.444081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7394.685835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7401.068917] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7403.175220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7407.483196] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7459.298395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.303404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.315216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.816409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.823533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.841158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.940617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.951784] Lustre: Skipped 1 previous similar message [ 7465.555860] Lustre: server umount lustre-OST0000 complete [ 7469.846167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7481.857403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7482.326593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.405791] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7489.522519] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7489.541918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7489.542131] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7489.553282] Lustre: Skipped 1 previous similar message [ 7494.373339] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7540.714484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.717967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7540.721203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7540.736637] Lustre: Skipped 1 previous similar message [ 7540.741634] LustreError: Skipped 1 previous similar message [ 7542.044327] Lustre: server umount lustre-OST0000 complete [ 7545.827147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7545.834040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.863572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.870675] Lustre: Skipped 2 previous similar messages [ 7556.076339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.084453] Lustre: Skipped 3 previous similar messages [ 7560.160157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7560.366857] Lustre: server umount lustre-MDT0000 complete [ 7564.134088] LustreError: 240657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779133990 with bad export cookie 1508308031350007767 [ 7564.136821] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.148700] LustreError: 240657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7564.549382] Lustre: server umount lustre-MDT0001 complete [ 7576.281438] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7579.199480] Key type lgssc unregistered [ 7579.536637] LNet: 245619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7579.552230] LNetError: 245619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7579.572815] LNet: Removed LNI 192.168.206.143@tcp [ 7580.562627] Key type .llcrypt unregistered [ 7580.564327] Key type ._llcrypt unregistered [ 7609.108497] Key type ._llcrypt registered [ 7609.111262] Key type .llcrypt registered [ 7609.224475] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7610.228364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7610.397494] alg: No test for adler32 (adler32-zlib) [ 7611.438777] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 7611.741306] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7613.496314] Key type lgssc registered [ 7614.710399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7622.710692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7622.745262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.235287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.590495] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7634.421649] LustreError: 247072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.441252] LustreError: 247072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7637.977838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.382889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7642.881166] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7648.865790] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7652.647523] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7659.758058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.114725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.785865] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7669.355395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7672.545847] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7699.425323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7699.438797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.460058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7700.453538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7700.461540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7700.481896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.825371] Lustre: server umount lustre-OST0000 complete [ 7710.186958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7710.200977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.221537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7710.232076] Lustre: Skipped 1 previous similar message [ 7712.033517] Lustre: server umount lustre-MDT0000 complete [ 7715.648606] LustreError: 248690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134141 with bad export cookie 17482114120447019756 [ 7715.652429] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7715.659519] LustreError: 248690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7715.987424] Lustre: server umount lustre-MDT0001 complete [ 7725.856498] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7728.422225] Key type lgssc unregistered [ 7728.721591] LNet: 251024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7728.729692] LNetError: 251024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7728.743612] LNet: Removed LNI 192.168.206.143@tcp [ 7729.535950] Key type .llcrypt unregistered [ 7729.538538] Key type ._llcrypt unregistered [ 7748.800517] Key type ._llcrypt registered [ 7748.815201] Key type .llcrypt registered [ 7748.958094] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7750.215587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7750.375718] alg: No test for adler32 (adler32-zlib) [ 7751.420626] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 7751.692140] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7753.352173] Key type lgssc registered [ 7754.685181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7763.159726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7763.178375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.753909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.414870] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7774.179840] LustreError: 252281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.194545] LustreError: 252281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7778.223934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.735063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.686805] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7788.378269] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7792.178352] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7799.367490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.710380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.744226] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7809.983585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7810.280608] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7824.354677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7824.379553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.414589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7825.387556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7825.388706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7825.408079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.610884] Lustre: server umount lustre-OST0000 complete [ 7835.111696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.117363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.126920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.132153] Lustre: Skipped 1 previous similar message [ 7838.599079] Lustre: server umount lustre-MDT0000 complete [ 7840.742594] LustreError: 252281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.767916] LustreError: 252281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7842.187420] LustreError: 252262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134268 with bad export cookie 8855105036628546780 [ 7842.192952] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.198171] LustreError: 252262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7842.541860] Lustre: server umount lustre-MDT0001 complete [ 7854.626260] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7857.319325] Key type lgssc unregistered [ 7857.609876] LNet: 255602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7857.621902] LNetError: 255602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7857.641856] LNet: Removed LNI 192.168.206.143@tcp [ 7858.396672] Key type .llcrypt unregistered [ 7858.405842] Key type ._llcrypt unregistered [ 7882.914590] Key type ._llcrypt registered [ 7882.923773] Key type .llcrypt registered [ 7883.061097] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7883.937191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7884.102944] alg: No test for adler32 (adler32-zlib) [ 7885.192167] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 7885.493231] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7887.184194] Key type lgssc registered [ 7888.119792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7895.563864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7895.584183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.163676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.030182] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7906.280435] LustreError: 257055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.289151] LustreError: 257055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7909.111793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.458388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7913.924890] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7919.718049] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7923.558738] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7929.798659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7930.121223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.032635] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7941.427886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7941.759660] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7970.273171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7970.286130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.298121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7972.321555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.324624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.326417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7974.844730] Lustre: server umount lustre-OST0000 complete [ 7981.024981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7981.043372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.069466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.072229] Lustre: Skipped 1 previous similar message [ 7984.410824] Lustre: server umount lustre-MDT0000 complete [ 7987.684922] LustreError: 257766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.709810] LustreError: 257766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7988.384130] LustreError: 257773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134414 with bad export cookie 16216708993766950951 [ 7988.389631] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.403419] LustreError: 257773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7988.913697] Lustre: server umount lustre-MDT0001 complete [ 7999.643612] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8003.076238] Key type lgssc unregistered [ 8003.410739] LNet: 261004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8003.422296] LNetError: 261004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8004.472986] LNet: Removed LNI 192.168.206.143@tcp [ 8005.752312] Key type .llcrypt unregistered [ 8005.756288] Key type ._llcrypt unregistered [ 8025.707921] Key type ._llcrypt registered [ 8025.711578] Key type .llcrypt registered [ 8025.836613] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8026.925988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.037171] alg: No test for adler32 (adler32-zlib) [ 8028.138750] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 8028.448451] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8030.144250] Key type lgssc registered [ 8031.551667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8039.879110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8039.899471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.532438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.540967] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8051.688570] LustreError: 262290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.707956] LustreError: 262290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8054.042788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.312333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8058.131712] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8063.399444] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8067.237972] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8074.538634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.993264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.925203] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8086.268276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8087.587743] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8100.837659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.857908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.883541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.859830] LustreError: 263909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.862347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.876867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8101.953572] Lustre: server umount lustre-OST0000 complete [ 8106.977821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.987649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8107.013799] Lustre: Skipped 1 previous similar message [ 8107.032620] Lustre: Skipped 1 previous similar message [ 8112.100417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.105782] Lustre: Skipped 1 previous similar message [ 8112.747901] Lustre: server umount lustre-MDT0000 complete [ 8115.649362] LustreError: 262255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134541 with bad export cookie 7830868993417827598 [ 8115.653996] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.660932] LustreError: 262255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8116.151935] Lustre: server umount lustre-MDT0001 complete [ 8125.640404] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8128.321842] Key type lgssc unregistered [ 8128.574283] LNet: 265635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8128.585756] LNetError: 265635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8128.605462] LNet: Removed LNI 192.168.206.143@tcp [ 8129.292091] Key type .llcrypt unregistered [ 8129.293961] Key type ._llcrypt unregistered [ 8139.076751] Key type ._llcrypt registered [ 8139.082128] Key type .llcrypt registered [ 8139.202867] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8154.493072] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8167.876891] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8168.776889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8168.874659] alg: No test for adler32 (adler32-zlib) [ 8169.904531] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 8170.100963] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8171.808201] Key type lgssc registered [ 8172.729351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8203.187903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8203.256856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.472253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8204.533963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8204.677210] Lustre: lustre-MDT0000: new disk, initializing [ 8204.750557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.769671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8208.987765] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8218.093289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.272698] Lustre: 269179:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8218.348591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8218.353379] Lustre: Skipped 1 previous similar message [ 8218.463987] Lustre: lustre-MDT0001: new disk, initializing [ 8218.601628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8218.642467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8218.656094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8222.696790] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8229.671404] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8233.976139] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8241.324996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.720777] Lustre: lustre-OST0000: new disk, initializing [ 8241.724828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8241.732260] Lustre: 270919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.898582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.406413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8248.411527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8248.466519] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8248.531859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8255.759391] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8259.943354] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8260.146692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8264.335602] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8264.641420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8269.300078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8269.303275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.315017] LustreError: Skipped 1 previous similar message [ 8269.320479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8269.360087] Lustre: Skipped 1 previous similar message [ 8274.412577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8274.420216] Lustre: Skipped 1 previous similar message [ 8279.525646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8279.533304] Lustre: Skipped 1 previous similar message [ 8281.056098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8281.420234] Lustre: server umount lustre-OST0000 complete [ 8289.774365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.786880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.795244] Lustre: Skipped 1 previous similar message [ 8289.817249] Lustre: Skipped 2 previous similar messages [ 8292.363108] Lustre: server umount lustre-MDT0000 complete [ 8296.478642] LustreError: 270918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134722 with bad export cookie 942479103465096061 [ 8296.480809] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.487957] LustreError: 270918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8297.007322] Lustre: server umount lustre-MDT0001 complete [ 8305.945746] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:05:29 (1779134729) [ 8313.659640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.095727] LustreError: 273189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.127323] LustreError: 273189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8314.200200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8319.307357] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8319.465117] LustreError: 273190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.579074] LustreError: 273189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.513684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.699482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8329.894657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8335.017631] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8342.869475] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8346.639722] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8354.455961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8354.794286] LustreError: 274782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.104783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8362.742514] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8368.622655] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8379.880926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8379.886797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.898547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8379.916917] Lustre: Skipped 1 previous similar message [ 8385.061888] Lustre: server umount lustre-OST0000 complete [ 8390.116532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.812958] Lustre: server umount lustre-MDT0000 complete [ 8395.238145] LustreError: 273186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.264221] LustreError: 273186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8398.692989] LustreError: 273170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779134824 with bad export cookie 942479103465096943 [ 8398.697830] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8398.707595] LustreError: 273170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8399.136744] Lustre: server umount lustre-MDT0001 complete [ 8410.390382] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8413.093537] Key type lgssc unregistered [ 8413.393426] LNet: 276371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8413.405725] LNetError: 276371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8413.429393] LNet: Removed LNI 192.168.206.143@tcp [ 8414.363203] Key type .llcrypt unregistered [ 8414.368952] Key type ._llcrypt unregistered [ 8437.662912] Key type ._llcrypt registered [ 8437.666385] Key type .llcrypt registered [ 8437.819264] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8452.238993] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8453.120365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8453.180061] alg: No test for adler32 (adler32-zlib) [ 8454.229796] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 8454.490324] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8456.170410] Key type lgssc registered [ 8457.353561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8505.649701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8505.670114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.985852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8507.039377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8507.182458] Lustre: lustre-MDT0000: new disk, initializing [ 8507.353256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.373940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8512.488528] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8524.321779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.444793] Lustre: 280215:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8524.486560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8524.490707] Lustre: Skipped 1 previous similar message [ 8524.574724] Lustre: lustre-MDT0001: new disk, initializing [ 8524.690036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8524.752304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8524.769252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8529.248621] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8536.873599] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8540.359302] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8549.008926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.297960] Lustre: lustre-OST2710: new disk, initializing [ 8549.303543] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8549.308287] Lustre: 281957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8549.403709] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8555.328905] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8556.603964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8556.612259] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8556.749416] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8562.206390] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8570.042503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.280947] Lustre: lustre-OST03e8: new disk, initializing [ 8570.284777] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8570.293539] Lustre: 282909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8570.413111] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8577.648917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8577.669923] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8577.906872] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8578.164259] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8585.535953] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8593.398928] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8593.608758] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8598.007124] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8598.251068] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8603.540382] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8603.778843] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8608.588216] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8608.855892] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8613.873069] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.886782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8613.903748] Lustre: Skipped 3 previous similar messages [ 8613.911451] Lustre: Skipped 2 previous similar messages [ 8615.630511] Lustre: server umount lustre-MDT0000 complete [ 8619.074742] LustreError: 280207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135044 with bad export cookie 14151866902276602319 [ 8619.094331] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.096031] LustreError: 280207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8619.448599] Lustre: server umount lustre-MDT0001 complete [ 8633.389708] Lustre: server umount lustre-OST2710 complete [ 8648.679526] Lustre: server umount lustre-OST03e8 complete [ 8669.813667] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8678.571934] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8710.060626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.300741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8710.320214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8710.377577] Lustre: lustre-MDT0000: new disk, initializing [ 8710.477204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8710.490449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8714.486691] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8724.417932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.557712] Lustre: 287954:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8724.605280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8724.610090] Lustre: Skipped 1 previous similar message [ 8724.703275] Lustre: lustre-MDT0001: new disk, initializing [ 8724.796125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8724.829883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8724.838826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8728.947247] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8736.858488] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8740.891816] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8748.158742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.423607] Lustre: lustre-OST0000: new disk, initializing [ 8748.428715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8748.436588] Lustre: 289696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.533834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.604746] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8755.827321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8755.838685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.051649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8762.108385] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8768.330917] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8768.632321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8773.864967] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8774.135954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8776.673303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8776.678828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.684244] LustreError: Skipped 1 previous similar message [ 8776.685056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.685061] Lustre: Skipped 1 previous similar message [ 8776.727304] Lustre: Skipped 1 previous similar message [ 8781.806880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8781.814180] Lustre: Skipped 2 previous similar messages [ 8782.687685] Lustre: server umount lustre-OST0000 complete [ 8786.401424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8786.411609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.427573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8792.047717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8792.054582] Lustre: Skipped 2 previous similar messages [ 8792.730330] Lustre: server umount lustre-MDT0000 complete [ 8796.572047] LustreError: 289695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135222 with bad export cookie 14151866902276603999 [ 8796.582365] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8796.585400] LustreError: 289695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8796.871521] Lustre: server umount lustre-MDT0001 complete [ 8805.236975] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:13:49 (1779135229) [ 8806.567842] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8808.233918] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:13:52 (1779135232) [ 8811.295588] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8844.739191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.271853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8845.308300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8845.572424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.112244] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8858.946680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8859.036807] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8859.067932] Lustre: 293400:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8863.028580] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8884.276301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.475830] Lustre: Denying initial registration attempt from nid 192.168.206.143@tcp, specified as failover [ 8884.490251] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8884.503910] LustreError: 294816:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8884.516582] LustreError: 294816:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8884.533796] LustreError: 294816:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8884.617507] Lustre: server umount lustre-OST0000 complete [ 8884.623914] LustreError: 294816:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8895.466578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8895.475291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8895.490779] Lustre: Skipped 1 previous similar message [ 8895.496437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8898.601282] Lustre: server umount lustre-MDT0000 complete [ 8901.090039] LustreError: 293405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.101606] LustreError: 293405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8902.186744] LustreError: 293390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135328 with bad export cookie 14151866902276604881 [ 8902.201229] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8902.212085] LustreError: 293390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8902.614274] Lustre: server umount lustre-MDT0001 complete [ 8912.504899] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8915.622965] Key type lgssc unregistered [ 8915.878812] LNet: 295919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8915.884406] LNetError: 295919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8915.895803] LNet: Removed LNI 192.168.206.143@tcp [ 8916.665113] Key type .llcrypt unregistered [ 8916.667961] Key type ._llcrypt unregistered [ 8927.820556] Key type ._llcrypt registered [ 8927.823509] Key type .llcrypt registered [ 8927.962904] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8929.197781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8929.382867] alg: No test for adler32 (adler32-zlib) [ 8930.361429] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 8930.544862] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8932.185839] Key type lgssc registered [ 8933.489170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8985.539954] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8997.251747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8997.277939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.874431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8998.964881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8999.250839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.867193] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9016.458828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.628137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9016.683483] Lustre: 298710:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9017.208059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.835639] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9030.962632] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9035.334098] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9057.040179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.370667] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9057.718567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9066.093070] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9072.770985] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9076.196672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9076.204829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.210631] LustreError: Skipped 1 previous similar message [ 9076.219824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9076.250895] Lustre: Skipped 1 previous similar message [ 9081.324582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9081.331547] Lustre: Skipped 1 previous similar message [ 9081.873578] Lustre: server umount lustre-OST0000 complete [ 9086.434927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.454777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9086.457582] Lustre: Skipped 1 previous similar message [ 9091.554741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9091.560911] Lustre: Skipped 2 previous similar messages [ 9092.572689] Lustre: server umount lustre-MDT0000 complete [ 9097.497063] LustreError: 298703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135523 with bad export cookie 5932372157853938271 [ 9097.507369] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9097.524466] LustreError: 298703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9098.232134] Lustre: server umount lustre-MDT0001 complete [ 9110.898716] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 9114.554895] Key type lgssc unregistered [ 9114.986806] LNet: 301930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9114.996741] LNetError: 301930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9115.016684] LNet: Removed LNI 192.168.206.143@tcp [ 9115.968316] Key type .llcrypt unregistered [ 9115.970989] Key type ._llcrypt unregistered [ 9146.321768] Key type ._llcrypt registered [ 9146.323953] Key type .llcrypt registered [ 9146.469797] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9148.223145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9148.269789] alg: No test for adler32 (adler32-zlib) [ 9149.440342] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 9149.826634] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 9151.512467] Key type lgssc registered [ 9152.878042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9161.720091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9161.735326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.242859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.760806] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9173.486730] LustreError: 303388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9173.509834] LustreError: 303388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9177.129184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.620522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9182.463902] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9189.612674] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9194.775599] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9204.974869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9205.478344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.247819] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9218.598673] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9236.817214] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 9238.498801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.500896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9238.531925] Lustre: Skipped 2 previous similar messages [ 9238.537068] Lustre: Skipped 1 previous similar message [ 9240.942434] Lustre: server umount lustre-MDT0000 complete [ 9241.916849] LustreError: 303384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.619289] LustreError: 303388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.644994] LustreError: 303388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9244.561058] LustreError: 303368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135670 with bad export cookie 12665128276057992531 [ 9244.577489] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.581665] LustreError: 303368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9247.025410] LustreError: 305152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.067149] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [ 9247.079770] Lustre: Skipped 1 previous similar message [ 9248.747631] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.770649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9249.779811] Lustre: Skipped 1 previous similar message [ 9251.075742] Lustre: server umount lustre-MDT0001 complete [ 9258.576814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9269.428496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.704548] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xafc3931079314ae9 [ 9289.734289] Lustre: MGC192.168.206.143@tcp: Connection restored to 0@lo (at 0@lo) [ 9290.348911] LustreError: 304993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9290.385263] LustreError: 304993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9290.462732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.509653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9296.100606] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9299.939135] LustreError: 306667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.965142] LustreError: 306667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9305.066966] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9308.606640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.194203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.294849] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9315.053926] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9322.386687] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9324.610815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9329.122749] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9331.157374] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9336.801896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9336.810870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9336.833171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9339.880260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.883864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9342.773259] Lustre: server umount lustre-OST0000 complete [ 9350.116217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.123971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9350.127619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9350.127627] Lustre: Skipped 2 previous similar messages [ 9350.138734] Lustre: Skipped 1 previous similar message [ 9361.376111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9361.800899] Lustre: server umount lustre-MDT0000 complete [ 9364.455720] LustreError: 306672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.477288] LustreError: 306672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9366.300788] LustreError: 303368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135792 with bad export cookie 12665128276058000105 [ 9366.308849] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9366.313449] LustreError: 303368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9366.677651] Lustre: server umount lustre-MDT0001 complete [ 9380.770320] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 9384.709921] Key type lgssc unregistered [ 9385.195877] LNet: 309102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9385.211233] LNetError: 309102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9385.259362] LNet: Removed LNI 192.168.206.143@tcp [ 9386.480666] Key type .llcrypt unregistered [ 9386.483789] Key type ._llcrypt unregistered [ 9416.375203] Key type ._llcrypt registered [ 9416.376892] Key type .llcrypt registered [ 9416.518910] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9417.612859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9417.627155] alg: No test for adler32 (adler32-zlib) [ 9418.744852] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 9419.127507] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 9420.856225] Key type lgssc registered [ 9422.159777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9430.787376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9430.816900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9432.428992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9436.498600] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9442.789763] LustreError: 310556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9442.824091] LustreError: 310556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9445.374778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.881020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9451.095908] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9457.307269] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9461.044990] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9464.259179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.263753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.281495] Lustre: Skipped 1 previous similar message [ 9469.409813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9469.416407] Lustre: Skipped 2 previous similar messages [ 9470.453482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9470.471127] Lustre: Skipped 1 previous similar message [ 9470.723757] Lustre: server umount lustre-MDT0000 complete [ 9474.323225] LustreError: 310537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779135900 with bad export cookie 4024795415953271100 [ 9474.330115] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9474.338270] LustreError: 310537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9474.617143] Lustre: server umount lustre-MDT0001 complete [ 9490.454642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.731552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9490.771252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9490.963149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9495.452957] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9505.435194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.599765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9505.638965] Lustre: 312917:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9505.939291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9510.788719] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9516.652278] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9520.739951] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9531.701883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9531.996991] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9532.011748] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9532.018367] LustreError: 314577:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9532.023575] LustreError: 314577:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9532.029210] LustreError: 314577:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9532.096953] Lustre: server umount lustre-OST0000 complete [ 9532.099502] LustreError: 314577:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9543.097847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.340228] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9543.486408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.965389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9549.137924] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9555.142632] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9564.614902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9564.717370] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9564.833373] Lustre: lustre-OST0001: new disk, initializing [ 9564.839174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9564.850208] Lustre: 316006:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9564.992201] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9571.624580] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9572.025136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9572.060925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9572.161424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9578.064893] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9582.566913] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9582.570918] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.574404] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9582.598941] Lustre: Skipped 1 previous similar message [ 9587.687933] LustreError: 315044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.709250] LustreError: 315044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9587.756126] Lustre: server umount lustre-OST0001 complete [ 9592.805683] LustreError: 316364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.811413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.825061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9592.825069] Lustre: Skipped 1 previous similar message [ 9592.826680] LustreError: 316364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9592.871197] Lustre: Skipped 1 previous similar message [ 9597.937395] LustreError: 315046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9597.977701] LustreError: 315046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9598.140672] Lustre: server umount lustre-OST0000 complete [ 9603.056767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9603.057275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9603.074763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9603.074775] Lustre: Skipped 3 previous similar messages [ 9603.115104] Lustre: Skipped 1 previous similar message [ 9608.660148] Lustre: server umount lustre-MDT0000 complete [ 9612.451559] LustreError: 313641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136038 with bad export cookie 4024795415953271653 [ 9612.452934] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9612.461484] LustreError: 313641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9612.732611] Lustre: server umount lustre-MDT0001 complete [ 9623.298511] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 9626.083588] Key type lgssc unregistered [ 9626.389736] LNet: 317834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9626.403388] LNetError: 317834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9626.430626] LNet: Removed LNI 192.168.206.143@tcp [ 9627.361665] Key type .llcrypt unregistered [ 9627.365209] Key type ._llcrypt unregistered [ 9699.228724] Key type ._llcrypt registered [ 9699.231419] Key type .llcrypt registered [ 9699.344450] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 9716.050483] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9717.128637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9717.271806] alg: No test for adler32 (adler32-zlib) [ 9718.305720] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 9718.545670] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 9720.312229] Key type lgssc registered [ 9721.540738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9758.188595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9758.248765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.528747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9759.561855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9759.667212] Lustre: lustre-MDT0000: new disk, initializing [ 9759.758357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9759.794242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9764.329443] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9776.210680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.351759] Lustre: 322792:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9776.423826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9776.427521] Lustre: Skipped 1 previous similar message [ 9776.508836] Lustre: lustre-MDT0001: new disk, initializing [ 9776.611613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9776.661179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9776.673171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9781.438733] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9789.821190] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9793.694435] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9800.646616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9801.028455] Lustre: lustre-OST0000: new disk, initializing [ 9801.038793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9801.047622] Lustre: 324534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9801.164716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.893155] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9810.538965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9810.553542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9810.697522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9816.587749] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9822.203743] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9822.513223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9828.010680] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9828.344457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9831.396806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9831.403197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.408150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9831.418513] Lustre: Skipped 1 previous similar message [ 9836.522647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9836.528369] Lustre: Skipped 2 previous similar messages [ 9837.039654] Lustre: server umount lustre-OST0000 complete [ 9841.640519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.677652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.693582] Lustre: Skipped 1 previous similar message [ 9846.753377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9846.762949] Lustre: Skipped 1 previous similar message [ 9848.280337] Lustre: server umount lustre-MDT0000 complete [ 9853.714737] LustreError: 322782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136279 with bad export cookie 4554460669315341658 [ 9853.719962] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9853.731739] LustreError: 322782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9854.525973] Lustre: server umount lustre-MDT0001 complete [ 9865.653329] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:31:29 (1779136289) [ 9873.808898] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:31:37 (1779136297) [ 9882.622863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.146356] LustreError: 327147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.165518] LustreError: 327147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9883.236209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.978518] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9888.239295] LustreError: 327148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.346131] LustreError: 327147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.436524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.831595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.427486] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9908.530528] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9913.041333] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9921.281718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9921.845143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9930.092425] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9937.379508] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9944.678109] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9946.600064] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9948.721149] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9950.977673] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9953.466801] Lustre: Failing over lustre-MDT0000 [ 9953.922073] Lustre: server umount lustre-MDT0000 complete [ 9954.276362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9954.282318] LustreError: Skipped 1 previous similar message [ 9954.285276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.295102] Lustre: Skipped 1 previous similar message [ 9954.299668] LustreError: 327148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.755525] LustreError: 327144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.782305] LustreError: 327144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9969.971918] LustreError: 327142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.004668] LustreError: 327142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9971.065324] Lustre: 320984:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779136380/real 1779136380] req@000000009eccb131 x1865559461400576/t0(0) o400->MGC192.168.206.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779136396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9971.119960] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.085826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.900616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9981.985055] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9986.355626] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9987.088110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9987.166363] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9987.265786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9988.110523] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9997.480825] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9998.959204] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10003.177923] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10005.218736] Lustre: Failing over lustre-MDT0000 [10005.604954] Lustre: server umount lustre-MDT0000 complete [10006.860983] LustreError: 328402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.888269] LustreError: 328402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [10007.535131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.548570] Lustre: Skipped 3 previous similar messages [10023.904135] Lustre: 320983:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779136433/real 1779136433] req@000000008348b210 x1865559461421696/t0(0) o400->MGC192.168.206.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779136449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10023.961272] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.347558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.147826] LustreError: 320980:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bdf10025 x1865559461427200/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10034.741557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10034.882301] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10038.583574] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10039.615978] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10039.800155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10039.804176] Lustre: Skipped 2 previous similar messages [10039.853142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10039.913736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10047.390878] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10049.207138] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10060.257464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10060.267797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.277664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10060.290653] Lustre: Skipped 2 previous similar messages [10060.303189] Lustre: Skipped 2 previous similar messages [10061.926908] Lustre: server umount lustre-MDT0000 complete [10065.377615] LustreError: 327147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.385955] LustreError: 327147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [10065.983480] LustreError: 327129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136491 with bad export cookie 4554460669315344374 [10065.992773] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10066.002670] LustreError: 327129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10066.643073] Lustre: server umount lustre-MDT0001 complete [10080.633603] Lustre: server umount lustre-OST0000 complete [10097.575810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.095271] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10112.366733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.918727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10112.933644] Lustre: Skipped 1 previous similar message [10118.256908] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10124.644444] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10129.422717] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10137.229881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10137.517826] LustreError: 334516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.557808] LustreError: 334516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10144.343084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10146.594593] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10153.472382] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10159.590988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10159.608791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.623322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10159.672188] Lustre: Skipped 2 previous similar messages [10161.443493] Lustre: server umount lustre-OST0000 complete [10169.317535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10180.064122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10180.066598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10180.097848] Lustre: Skipped 6 previous similar messages [10180.292784] Lustre: server umount lustre-MDT0000 complete [10184.498906] LustreError: 332903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136610 with bad export cookie 4554460669315345116 [10184.510658] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10184.926352] Lustre: server umount lustre-MDT0001 complete [10197.403471] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [10200.997524] Key type lgssc unregistered [10201.401212] LNet: 336100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10201.405550] LNetError: 336100:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10201.417258] LNet: Removed LNI 192.168.206.143@tcp [10202.386473] Key type .llcrypt unregistered [10202.390749] Key type ._llcrypt unregistered [10228.634646] Key type ._llcrypt registered [10228.638704] Key type .llcrypt registered [10228.766492] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [10229.896279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10229.989951] alg: No test for adler32 (adler32-zlib) [10231.119406] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [10231.324273] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [10232.952490] Key type lgssc registered [10234.103141] Lustre: Echo OBD driver; http://www.lustre.org/ [10243.593208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10243.629507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.436096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.468740] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10255.875413] LustreError: 337653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.896153] LustreError: 337653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10260.343551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.794457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10266.048680] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10273.958535] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10279.485524] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10288.223988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.948287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10291.023466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10297.334935] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10304.294909] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10320.884056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.908243] Lustre: Skipped 1 previous similar message [10320.908624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.924578] Lustre: Skipped 2 previous similar messages [10324.413397] Lustre: server umount lustre-MDT0000 complete [10325.989361] LustreError: 337653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.011158] LustreError: 337653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10328.951525] LustreError: 337635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136754 with bad export cookie 4651209054847134627 [10328.953611] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.964935] LustreError: 337635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10329.386825] Lustre: server umount lustre-MDT0001 complete [10343.770439] Lustre: server umount lustre-OST0000 complete [10358.711103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.322909] LustreError: 341051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.446949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10364.307713] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10364.914862] LustreError: 341052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.021633] LustreError: 341051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10374.580805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10375.144996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10379.221540] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10385.539219] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10389.820431] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10397.667984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10398.065251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10404.211087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10404.529175] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10410.690272] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10424.803455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.806289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.814870] Lustre: Skipped 2 previous similar messages [10424.821809] Lustre: Skipped 2 previous similar messages [10429.373652] Lustre: server umount lustre-MDT0000 complete [10429.925568] LustreError: 341052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.940086] LustreError: 341052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10434.072482] LustreError: 341033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136859 with bad export cookie 4651209054847135740 [10434.073277] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10434.085437] LustreError: 341033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10434.609240] Lustre: server umount lustre-MDT0001 complete [10449.995259] Lustre: server umount lustre-OST0000 complete [10465.860528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.389836] LustreError: 344625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10466.454726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.317689] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10480.905560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.112280] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10492.009769] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10496.117844] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10504.477024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10504.927560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10504.939199] Lustre: Skipped 1 previous similar message [10511.643253] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10512.121576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10518.975578] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10527.748866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10527.765408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.781124] Lustre: Skipped 1 previous similar message [10527.787755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10527.792775] Lustre: Skipped 1 previous similar message [10530.295273] Lustre: server umount lustre-OST0000 complete [10537.449812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10537.466791] LustreError: Skipped 1 previous similar message [10537.476767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.503057] Lustre: Skipped 1 previous similar message [10537.506547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10543.074505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10543.082599] Lustre: Skipped 2 previous similar messages [10548.192134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10548.343193] Lustre: server umount lustre-MDT0000 complete [10551.832936] LustreError: 344605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779136977 with bad export cookie 4651209054847136762 [10551.835056] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10551.847885] LustreError: 344605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10552.131600] Lustre: server umount lustre-MDT0001 complete [10562.024281] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [10564.748575] Key type lgssc unregistered [10565.018376] LNet: 347802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10565.029271] LNetError: 347802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10566.056078] LNet: Removed LNI 192.168.206.143@tcp [10566.840789] Key type .llcrypt unregistered [10566.847606] Key type ._llcrypt unregistered [10597.033850] Key type ._llcrypt registered [10597.038490] Key type .llcrypt registered [10597.139971] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [10597.917127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10598.045519] alg: No test for adler32 (adler32-zlib) [10599.107779] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [10599.403099] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [10601.048206] Key type lgssc registered [10602.136855] Lustre: Echo OBD driver; http://www.lustre.org/ [10611.807650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10611.852653] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.867782] LustreError: 349279:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10611.901803] LustreError: 349279:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10611.918872] LustreError: 349279:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10611.933949] LustreError: 349279:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10611.945192] LustreError: 349279:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10620.374056] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.396957] LustreError: 349676:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10620.423685] LustreError: 349676:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10620.440406] LustreError: 349676:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10620.458873] LustreError: 349676:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10620.468489] LustreError: 349676:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10641.492572] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [10645.359936] Key type lgssc unregistered [10645.771885] LNet: 350664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10645.781623] LNetError: 350664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10645.797818] LNet: Removed LNI 192.168.206.143@tcp [10646.727669] Key type .llcrypt unregistered [10646.729895] Key type ._llcrypt unregistered [10668.170302] Key type ._llcrypt registered [10668.175026] Key type .llcrypt registered [10668.336956] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [10685.593779] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [10687.654823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10687.691652] alg: No test for adler32 (adler32-zlib) [10688.832657] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [10689.215396] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [10690.984720] Key type lgssc registered [10692.880671] Lustre: Echo OBD driver; http://www.lustre.org/ [10733.330987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10733.371208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.585794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10734.608915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10734.674760] Lustre: lustre-MDT0000: new disk, initializing [10734.742806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10734.758057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10738.357760] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10751.569817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.663656] Lustre: 353842:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.757961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10751.762231] Lustre: Skipped 1 previous similar message [10751.832633] Lustre: lustre-MDT0001: new disk, initializing [10751.961399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10752.032807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10752.038511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10758.191358] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10768.618271] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10772.889692] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10780.580869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.033676] Lustre: lustre-OST0000: new disk, initializing [10781.038342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10781.045791] Lustre: 355588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10781.143325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10787.387595] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10790.457760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10790.471180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10790.601342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10795.679541] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10800.424396] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10800.677230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10805.303481] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10805.548793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10809.321657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.345903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10809.351261] Lustre: Skipped 1 previous similar message [10813.736919] Lustre: server umount lustre-OST0000 complete [10818.538001] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10818.552486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.592651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.605648] Lustre: Skipped 1 previous similar message [10819.559762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.674362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10824.693028] Lustre: Skipped 3 previous similar messages [10829.796903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10829.806253] Lustre: Skipped 1 previous similar message [10832.352154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10832.522602] Lustre: server umount lustre-MDT0000 complete [10836.419187] LustreError: 353835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137262 with bad export cookie 12222188356181761240 [10836.421796] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10836.435505] LustreError: 353835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10836.720605] Lustre: server umount lustre-MDT0001 complete [10845.499850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10847.122875] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:47:51 (1779137271) [10854.602627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.136830] LustreError: 357903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.157985] LustreError: 357903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10855.233936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10859.794953] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10860.515617] LustreError: 357904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.634602] LustreError: 357903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.468873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10870.182536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10876.257939] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10883.292762] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10887.541345] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10894.517702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.762791] LustreError: 359498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.003099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10902.343957] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10909.675517] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10917.471841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.666558] Lustre: lustre-OST0001: new disk, initializing [10917.671187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10917.678462] Lustre: 360401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10917.804234] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10924.022567] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10925.677179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10925.703850] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10925.814029] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10932.368185] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10936.300131] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10936.301839] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.318879] LustreError: Skipped 1 previous similar message [10936.321609] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10936.344565] Lustre: Skipped 1 previous similar message [10941.914163] Lustre: server umount lustre-OST0001 complete [10944.312573] LustreError: 359499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.792673] LustreError: 359498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.823181] LustreError: 359498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10961.903290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10961.907524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10961.916391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10961.925857] Lustre: Skipped 1 previous similar message [10961.939944] Lustre: Skipped 5 previous similar messages [10964.483410] Lustre: server umount lustre-OST0000 complete [10972.132696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10981.354750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10981.357911] Lustre: Skipped 5 previous similar messages [10982.369960] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10982.716464] Lustre: server umount lustre-MDT0000 complete [10986.471779] LustreError: 358614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10986.494444] LustreError: 358614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10987.669616] LustreError: 357885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137413 with bad export cookie 12222188356181762122 [10987.683901] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.687336] LustreError: 357885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11002.336151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11002.674811] Lustre: server umount lustre-MDT0001 complete [11015.682912] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11018.874178] Key type lgssc unregistered [11019.244526] LNet: 362262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11019.248571] LNetError: 362262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11019.266926] LNet: Removed LNI 192.168.206.143@tcp [11020.400316] Key type .llcrypt unregistered [11020.403659] Key type ._llcrypt unregistered [11072.839325] Key type ._llcrypt registered [11072.845636] Key type .llcrypt registered [11073.012903] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11074.341736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11074.394060] alg: No test for adler32 (adler32-zlib) [11075.422925] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [11075.722991] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11077.387228] Key type lgssc registered [11078.405308] Lustre: Echo OBD driver; http://www.lustre.org/ [11086.129682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11086.152138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.449537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11087.471383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11087.705847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11092.032582] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11101.599193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.683153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11101.722766] Lustre: 365100:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11102.077459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11106.806078] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11112.387631] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11116.212966] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11119.078962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.107598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11119.116468] Lustre: Skipped 1 previous similar message [11124.194337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11124.203322] Lustre: Skipped 1 previous similar message [11124.723948] Lustre: server umount lustre-MDT0000 complete [11128.315866] LustreError: 365090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137554 with bad export cookie 12402126489014586240 [11128.334433] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.343422] LustreError: 365090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11128.610312] Lustre: server umount lustre-MDT0001 complete [11137.543297] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:52:41 (1779137561) [11143.711357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.264303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11149.134548] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11149.282862] LustreError: 367384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.306283] LustreError: 367384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11154.411716] LustreError: 367385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.781602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.151304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11162.750795] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11169.001645] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11172.789525] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11178.974841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11179.192411] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11179.395779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11185.753417] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11191.529975] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11199.909731] Lustre: Permanently deactivating lustre-OST0000 [11199.941583] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11207.183957] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11209.046745] LustreError: 370029:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11209.054294] LustreError: 370029:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11209.058259] LustreError: 370029:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11212.092325] LustreError: 370077:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11212.106377] LustreError: 370077:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11220.313699] Lustre: server umount lustre-OST0000 complete [11222.821927] LustreError: 370326:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11222.829758] LustreError: 370326:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11228.647224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11228.669280] Lustre: Skipped 1 previous similar message [11228.674165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11228.677979] Lustre: Skipped 1 previous similar message [11233.763506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.778547] Lustre: Skipped 1 previous similar message [11234.846330] Lustre: server umount lustre-MDT0000 complete [11238.706744] LustreError: 367366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137664 with bad export cookie 12402126489014586821 [11238.711395] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11238.724068] LustreError: 367366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11249.376192] Lustre: server umount lustre-MDT0001 complete [11263.470614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11269.177665] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11272.660114] LustreError: 371472:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11279.261401] Lustre: 371812:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.143@tcp [11281.405278] Lustre: server umount MGS complete [11292.605089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.062279] LustreError: 372538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.087975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11293.095945] Lustre: Skipped 1 previous similar message [11293.149966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11297.178808] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11298.280385] LustreError: 372539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.403986] LustreError: 372538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.514573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.913848] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11310.175684] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11315.463725] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11319.197489] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11321.075706] Lustre: Permanently reactivating lustre-OST0000 [11321.109065] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11321.117354] Lustre: Skipped 2 previous similar messages [11348.448343] LustreError: 374093:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11355.781299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.083801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11356.088582] Lustre: Skipped 1 previous similar message [11362.550920] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11368.148160] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11372.516806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11372.533925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.552200] Lustre: Skipped 1 previous similar message [11372.563834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11373.555271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11378.634970] Lustre: server umount lustre-OST0000 complete [11382.753212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11382.779621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11382.796979] Lustre: Skipped 1 previous similar message [11382.814674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11382.826162] Lustre: Skipped 2 previous similar messages [11388.812438] Lustre: server umount lustre-MDT0000 complete [11388.900617] LustreError: 373249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11388.910578] LustreError: 373249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11392.579326] LustreError: 373886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137818 with bad export cookie 12402126489014587864 [11392.590764] LustreError: 373886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11392.594244] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11392.913427] Lustre: server umount lustre-MDT0001 complete [11404.476456] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11407.498106] Key type lgssc unregistered [11407.776840] LNet: 376544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11407.788865] LNetError: 376544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11407.804573] LNet: Removed LNI 192.168.206.143@tcp [11408.537520] Key type .llcrypt unregistered [11408.539870] Key type ._llcrypt unregistered [11427.221915] Key type ._llcrypt registered [11427.223824] Key type .llcrypt registered [11427.313553] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [11441.550696] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11442.404546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11442.467704] alg: No test for adler32 (adler32-zlib) [11443.466870] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [11443.695751] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11445.376317] Key type lgssc registered [11446.425120] Lustre: Echo OBD driver; http://www.lustre.org/ [11481.318811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11481.351757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.662227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11482.709765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11482.871547] Lustre: lustre-MDT0000: new disk, initializing [11483.020776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11483.042241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11487.275578] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11497.579545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11497.722111] Lustre: 379722:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11497.825881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11497.829756] Lustre: Skipped 1 previous similar message [11497.934875] Lustre: lustre-MDT0001: new disk, initializing [11498.035420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11498.064685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11498.071525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11502.180553] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11509.018238] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11512.749314] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11519.953877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11520.253345] Lustre: lustre-OST0000: new disk, initializing [11520.259753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11520.268320] Lustre: 381462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11520.356891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11526.412597] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11527.264838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11527.290463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11527.443621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11534.370782] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11538.679848] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11538.917136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11543.501559] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11543.707809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11548.129766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11548.131982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.138533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11548.161529] Lustre: Skipped 1 previous similar message [11551.912401] Lustre: server umount lustre-OST0000 complete [11558.371420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.385527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11558.391288] Lustre: Skipped 2 previous similar messages [11561.632457] Lustre: server umount lustre-MDT0000 complete [11565.395594] LustreError: 379715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779137991 with bad export cookie 16048146436516727753 [11565.397636] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11565.414891] LustreError: 379715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11565.946104] Lustre: server umount lustre-MDT0001 complete [11574.754901] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11576.584503] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:00:00 (1779138000) [11584.180047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11584.622848] LustreError: 383780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.648266] LustreError: 383780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11584.718654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.600339] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11590.115499] LustreError: 383781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.215780] LustreError: 383780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.271636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.334790] LustreError: 383781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.588230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11604.578415] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11611.206408] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11616.108963] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11623.449527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11623.946834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11630.441227] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11636.791525] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11638.992701] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11641.525647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11650.529869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.531048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11650.541222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11650.550714] Lustre: Skipped 2 previous similar messages [11653.239620] Lustre: server umount lustre-OST0000 complete [11660.777068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.784134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11660.792127] Lustre: Skipped 1 previous similar message [11660.798276] Lustre: Skipped 1 previous similar message [11663.448364] Lustre: server umount lustre-MDT0000 complete [11665.892718] LustreError: 383781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.914483] LustreError: 383781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11667.625563] LustreError: 383760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779138093 with bad export cookie 16048146436516728635 [11667.631597] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.652041] LustreError: 383760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11667.970589] Lustre: server umount lustre-MDT0001 complete [11680.313604] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11684.371356] Key type lgssc unregistered [11684.799670] LNet: 387158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.810710] LNetError: 387158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.834377] LNet: Removed LNI 192.168.206.143@tcp [11685.774115] Key type .llcrypt unregistered [11685.782595] Key type ._llcrypt unregistered [11714.956519] Key type ._llcrypt registered [11714.958522] Key type .llcrypt registered [11715.040233] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11715.370834] Key type .llcrypt unregistered [11715.373300] Key type ._llcrypt unregistered [11735.238154] Key type ._llcrypt registered [11735.248693] Key type .llcrypt registered [11735.429037] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11737.224897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11737.294959] alg: No test for adler32 (adler32-zlib) [11738.547319] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [11738.915821] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11740.672170] Key type lgssc registered [11742.411598] Lustre: Echo OBD driver; http://www.lustre.org/ [11752.603646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11752.628996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.267858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11759.673816] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11764.709426] LustreError: 389414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.740206] LustreError: 389414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11768.329613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.765471] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.790794] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11768.809551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11773.923862] LustreError: 389408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.944089] LustreError: 389408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11775.861471] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11779.045491] LustreError: 389414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11779.071305] LustreError: 389414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11780.088733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11782.359896] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11785.192498] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11785.217529] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11791.458180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.972264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11797.344454] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11807.211711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.220520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11807.228883] Lustre: Skipped 1 previous similar message [11807.233652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11807.260796] Lustre: Skipped 1 previous similar message [11812.324732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11812.344431] Lustre: Skipped 1 previous similar message [11812.825572] Lustre: server umount lustre-OST0000 complete [11822.563201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11822.564768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.578480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.598563] Lustre: Skipped 1 previous similar message [11822.610045] Lustre: Skipped 1 previous similar message [11824.045341] Lustre: server umount lustre-MDT0000 complete [11827.688391] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.704459] LustreError: 389413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11828.176204] LustreError: 389393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779138254 with bad export cookie 9761992109560001994 [11828.187732] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11828.189191] LustreError: 389393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11828.775757] Lustre: server umount lustre-MDT0001 complete [11840.423453] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11843.522225] Key type lgssc unregistered [11843.865639] LNet: 392397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11843.886708] LNetError: 392397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11843.924134] LNet: Removed LNI 192.168.206.143@tcp [11844.774851] Key type .llcrypt unregistered [11844.781708] Key type ._llcrypt unregistered [11871.910549] Key type ._llcrypt registered [11871.914374] Key type .llcrypt registered [11872.030933] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11873.623733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11873.690280] alg: No test for adler32 (adler32-zlib) [11874.905032] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [11875.190255] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11876.921420] Key type lgssc registered [11878.372445] Lustre: Echo OBD driver; http://www.lustre.org/ [11887.689870] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11887.705314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.268153] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11908.384198] LustreError: 393825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.143@tcp: failed processing log, type 4: rc = -110 [11935.073958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11942.275803] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11948.898496] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11956.167779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.033217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11957.155706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11962.094371] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11962.340209] LustreError: 393851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.359687] LustreError: 393851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11967.463319] LustreError: 393850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.510333] LustreError: 394750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.525135] LustreError: 394750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11971.388473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.789389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11977.255743] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11985.530102] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11990.394463] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12002.795986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12002.803338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.808772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.837036] Lustre: Skipped 1 previous similar message [12006.114534] Lustre: server umount lustre-OST0000 complete [12012.732204] Lustre: server umount lustre-MDT0000 complete [12013.035231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12013.059188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12013.080759] LustreError: 394753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12013.107596] LustreError: 394753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12017.231251] LustreError: 393833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779138443 with bad export cookie 27940661977933894 [12017.258357] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.937444] Lustre: server umount lustre-MDT0001 complete [12031.545551] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12035.574167] Key type lgssc unregistered [12036.002243] LNet: 397037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12036.009410] LNetError: 397037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12036.037843] LNet: Removed LNI 192.168.206.143@tcp [12037.210706] Key type .llcrypt unregistered [12037.219938] Key type ._llcrypt unregistered [12067.610914] Key type ._llcrypt registered [12067.613313] Key type .llcrypt registered [12067.770403] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12068.933367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12069.018831] alg: No test for adler32 (adler32-zlib) [12070.088979] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [12070.323874] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [12072.076082] Key type lgssc registered [12073.269615] Lustre: Echo OBD driver; http://www.lustre.org/ [12082.746844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12082.761813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.393503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12089.053825] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12094.948618] LustreError: 398496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.980767] LustreError: 398496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12099.491225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.874228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.032964] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12111.967384] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12116.286562] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12123.401457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12123.688795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.836945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12129.928635] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12135.514916] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12140.009095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.018340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12140.026443] Lustre: Skipped 2 previous similar messages [12140.049162] Lustre: Skipped 2 previous similar messages [12142.439545] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [12145.679734] Lustre: server umount lustre-MDT0000 complete [12147.519672] LustreError: 398491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.003333] LustreError: 398476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.43@tcp arrived at 1779138574 with bad export cookie 3104817717345356420 [12149.018684] LustreError: 398476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12151.567086] LustreError: 400112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779138577 with bad export cookie 3104817717345355524 [12151.576326] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12153.313576] LustreError: 399207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12153.314224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12153.331560] LustreError: 399207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12153.378850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12153.386425] Lustre: Skipped 3 previous similar messages [12156.328250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12156.329529] LustreError: 398493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.383452] LustreError: 398493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12157.823538] Lustre: server umount lustre-OST0000 complete [12161.516372] LustreError: 398492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.542486] LustreError: 398492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12165.373979] Lustre: server umount lustre-MDT0001 complete [12176.665559] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12179.222526] Key type lgssc unregistered [12179.662239] LNet: 401821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12179.686473] LNetError: 401821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12179.704833] LNet: Removed LNI 192.168.206.143@tcp [12180.635670] Key type .llcrypt unregistered [12180.640207] Key type ._llcrypt unregistered [12207.821455] Key type ._llcrypt registered [12207.827967] Key type .llcrypt registered [12207.974648] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12209.105579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12209.193245] alg: No test for adler32 (adler32-zlib) [12210.447638] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [12210.728542] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [12212.464164] Key type lgssc registered [12213.759631] Lustre: Echo OBD driver; http://www.lustre.org/ [12223.505579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12223.530873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.224932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12229.563394] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12235.750488] LustreError: 403287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12235.785462] LustreError: 403287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12238.186790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.508944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12243.437821] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12249.229671] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12253.531491] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12260.913255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12261.286505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12267.526476] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12270.525434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12273.818941] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12279.778432] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12279.793152] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.807701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12280.658610] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [12280.809153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.986940] Lustre: server umount lustre-MDT0001 complete [12285.746476] LustreError: 404650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.042083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12291.045044] LustreError: 403288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.057263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.066027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12291.066036] Lustre: Skipped 2 previous similar messages [12291.139233] LustreError: 403288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12296.166050] LustreError: 403287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.167563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.987338] Lustre: server umount lustre-OST0000 complete [12300.688811] Lustre: server umount lustre-MDT0000 complete [12314.527729] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12317.169277] Key type lgssc unregistered [12317.511683] LNet: 406614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12317.532099] LNetError: 406614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12317.555681] LNet: Removed LNI 192.168.206.143@tcp [12318.409824] Key type .llcrypt unregistered [12318.411964] Key type ._llcrypt unregistered [12344.374183] Key type ._llcrypt registered [12344.376020] Key type .llcrypt registered [12344.449524] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [12359.928876] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12361.108619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12361.169928] alg: No test for adler32 (adler32-zlib) [12362.401595] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [12362.712842] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [12364.440205] Key type lgssc registered [12366.250696] Lustre: Echo OBD driver; http://www.lustre.org/ [12411.075815] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12430.490393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12430.530171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.832427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12431.892774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12432.028795] Lustre: lustre-MDT0000: new disk, initializing [12432.158310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12432.178925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12436.662115] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12453.041373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12453.323600] Lustre: lustre-OST0000: new disk, initializing [12453.327567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12453.332595] Lustre: Skipped 1 previous similar message [12453.337716] Lustre: 411647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12453.432992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12458.622074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12458.644760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12458.697119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12460.427205] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12468.464188] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12483.805545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12483.938047] Lustre: 410647:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12484.030467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12484.132167] Lustre: lustre-MDT0001: new disk, initializing [12484.277316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12484.317527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12484.331575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12488.390713] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12497.865829] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12498.175742] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12502.722630] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12502.934619] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12509.667948] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.673133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12509.694284] Lustre: Skipped 1 previous similar message [12509.705972] Lustre: Skipped 1 previous similar message [12510.544292] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [12514.621523] Lustre: server umount lustre-MDT0001 complete [12519.910381] LustreError: 410659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.934828] LustreError: 410659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12525.044125] LustreError: 410658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.061018] LustreError: 410658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12530.148799] LustreError: 410655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.175883] LustreError: 410655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12535.267604] LustreError: 410659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.277724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12535.305490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12535.322879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.945687] Lustre: server umount lustre-OST0000 complete [12540.385181] LustreError: 410658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.127693] Lustre: server umount lustre-MDT0000 complete [12564.446963] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12567.523865] Key type lgssc unregistered [12567.819274] LNet: 415213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12567.825241] LNetError: 415213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12567.839224] LNet: Removed LNI 192.168.206.143@tcp [12568.748125] Key type .llcrypt unregistered [12568.752245] Key type ._llcrypt unregistered [12614.302474] Key type ._llcrypt registered [12614.306251] Key type .llcrypt registered [12614.445115] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12614.751256] Key type .llcrypt unregistered [12614.756549] Key type ._llcrypt unregistered [12638.577552] Key type ._llcrypt registered [12638.581232] Key type .llcrypt registered [12638.757735] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12640.984934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12641.189064] alg: No test for adler32 (adler32-zlib) [12642.443110] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [12642.837704] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [12644.736250] Key type lgssc registered [12646.144797] Lustre: Echo OBD driver; http://www.lustre.org/ [12679.732085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12679.775449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12681.028759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12681.115336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12681.234829] Lustre: lustre-MDT0000: new disk, initializing [12681.351353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12681.378061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12686.567646] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12698.252615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12698.388805] Lustre: 418618:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12698.465736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12698.468799] Lustre: Skipped 1 previous similar message [12698.545839] Lustre: lustre-MDT0001: new disk, initializing [12698.681333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12698.739556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12698.755934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12703.455545] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12711.702326] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12716.217688] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12724.494278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12724.799943] Lustre: lustre-OST0000: new disk, initializing [12724.804477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12724.810061] Lustre: 420360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12724.908354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12732.101101] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12732.478121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12732.488611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12732.627630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12740.523403] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12749.793754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12749.819630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12749.845891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.381754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.384759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.389571] Lustre: Skipped 1 previous similar message [12753.438129] Lustre: Skipped 1 previous similar message [12755.493664] Lustre: server umount lustre-MDT0000 complete [12760.908681] LustreError: 418609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779139186 with bad export cookie 9524779517501748243 [12760.914449] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12760.933199] LustreError: 418609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12761.581742] Lustre: server umount lustre-MDT0001 complete [12777.739890] Lustre: server umount lustre-OST0000 complete [12791.128953] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:20:14 (1779139214) [12800.720179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.168228] LustreError: 422279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12801.190257] LustreError: 422279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12801.275850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12805.852751] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12806.629258] LustreError: 422280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.763397] LustreError: 422279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.926526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12817.259135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12823.248500] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12829.873389] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12834.432265] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12854.891233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12855.099770] Lustre: Found index 0 for lustre-OST0000, updating log [12855.254835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12861.345847] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12867.832848] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12873.720210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12889.061687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12889.065087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12889.086760] Lustre: Skipped 1 previous similar message [12889.096688] Lustre: Skipped 1 previous similar message [12890.152043] LustreError: 423882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.190450] LustreError: 423882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12890.297864] Lustre: server umount lustre-OST0000 complete [12909.542503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12909.557689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12909.573310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.715413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.728060] Lustre: Skipped 3 previous similar messages [12918.368227] LNet: 1 local NIs in recovery (showing 1): 192.168.206.143@tcp [12921.313211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12921.554364] Lustre: server umount lustre-MDT0000 complete [12922.853405] LustreError: 422280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.864233] LustreError: 422280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12925.366720] LustreError: 422998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779139351 with bad export cookie 9524779517501749636 [12925.372281] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12925.382720] LustreError: 422998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12925.761625] Lustre: server umount lustre-MDT0001 complete [12935.065138] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:22:38 (1779139358) [13564.115039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13564.720320] LustreError: 428377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13564.876243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13569.703344] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [13570.019945] LustreError: 428378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.147659] LustreError: 428377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.391902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.813734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13586.204589] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [13593.141294] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13598.110255] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13619.057437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13619.391950] Lustre: Found index 0 for lustre-OST0000, updating log [13619.751156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13622.880187] LNet: 1 local NIs in recovery (showing 1): 192.168.206.143@tcp [13627.695775] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [13634.520778] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13637.194450] Lustre: server umount lustre-OST0000 complete [13642.211053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13642.223473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13642.232995] Lustre: Skipped 1 previous similar message [13642.238923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13642.246601] Lustre: Skipped 2 previous similar messages [13643.311219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13643.315106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.413392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13645.427681] Lustre: Skipped 4 previous similar messages [13647.739492] Lustre: server umount lustre-MDT0000 complete [13648.482953] LustreError: 429089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13648.498523] LustreError: 429089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13651.781533] LustreError: 430006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779140077 with bad export cookie 9524779517501750721 [13651.790595] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13651.801762] LustreError: 430006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13652.302175] Lustre: server umount lustre-MDT0001 complete [13682.273333] LNet: 1 local NIs in recovery (showing 1): 192.168.206.143@tcp [13687.273498] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [13699.218661] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [13748.941576] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:36:12 (1779140172) [13769.314345] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [13773.047771] Key type lgssc unregistered [13773.457710] LNet: 434618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13773.467107] LNetError: 434618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13883.053847] LNet: Removed LNI 192.168.206.143@tcp [13884.093122] Key type .llcrypt unregistered [13884.105156] Key type ._llcrypt unregistered [13902.433463] Key type ._llcrypt registered [13902.440189] Key type .llcrypt registered [13902.599272] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [13903.918735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13904.045725] alg: No test for adler32 (adler32-zlib) [13905.242977] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [13905.504150] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [13907.185244] Key type lgssc registered [13908.486621] Lustre: Echo OBD driver; http://www.lustre.org/ [13914.116941] LNet: Added LNI 192.168.206.143@tcp100 [8/256/0/180] [13915.128869] LNet: Added LNI 192.168.206.143@tcp101 [8/256/0/180] [13916.236883] LNet: Added LNI 192.168.206.143@tcp102 [8/256/0/180] [13917.317300] LNet: Added LNI 192.168.206.143@tcp103 [8/256/0/180] [13918.453027] LNet: Added LNI 192.168.206.143@tcp104 [8/256/0/180] [13919.698474] LNet: Added LNI 192.168.206.143@tcp105 [8/256/0/180] [13920.751607] LNet: Added LNI 192.168.206.143@tcp106 [8/256/0/180] [13921.972093] LNet: Added LNI 192.168.206.143@tcp107 [8/256/0/180] [13923.088401] LNet: Added LNI 192.168.206.143@tcp108 [8/256/0/180] [13924.206782] LNet: Added LNI 192.168.206.143@tcp109 [8/256/0/180] [13925.426903] LNet: Added LNI 192.168.206.143@tcp110 [8/256/0/180] [13926.256766] LNet: Added LNI 192.168.206.143@tcp111 [8/256/0/180] [13927.321217] LNet: Added LNI 192.168.206.143@tcp112 [8/256/0/180] [13928.370761] LNet: Added LNI 192.168.206.143@tcp113 [8/256/0/180] [13929.425748] LNet: Added LNI 192.168.206.143@tcp114 [8/256/0/180] [13930.612666] LNet: Added LNI 192.168.206.143@tcp115 [8/256/0/180] [13931.570078] LNet: Added LNI 192.168.206.143@tcp116 [8/256/0/180] [13932.695879] LNet: Added LNI 192.168.206.143@tcp117 [8/256/0/180] [13933.770901] LNet: Added LNI 192.168.206.143@tcp118 [8/256/0/180] [13935.255068] LNet: Added LNI 192.168.206.143@tcp119 [8/256/0/180] [13936.285816] LNet: Added LNI 192.168.206.143@tcp120 [8/256/0/180] [13937.408450] LNet: Added LNI 192.168.206.143@tcp121 [8/256/0/180] [13938.412829] LNet: Added LNI 192.168.206.143@tcp122 [8/256/0/180] [13939.658620] LNet: Added LNI 192.168.206.143@tcp123 [8/256/0/180] [13940.629858] LNet: Added LNI 192.168.206.143@tcp124 [8/256/0/180] [13941.870308] LNet: Added LNI 192.168.206.143@tcp125 [8/256/0/180] [13943.024165] LNet: Added LNI 192.168.206.143@tcp126 [8/256/0/180] [13944.162416] LNet: Added LNI 192.168.206.143@tcp127 [8/256/0/180] [13945.317887] LNet: Added LNI 192.168.206.143@tcp128 [8/256/0/180] [13946.564883] LNet: Added LNI 192.168.206.143@tcp129 [8/256/0/180] [13947.724367] LNet: Added LNI 192.168.206.143@tcp130 [8/256/0/180] [13948.718803] LNet: Added LNI 192.168.206.143@tcp131 [8/256/0/180] [13949.686772] LNet: Added LNI 192.168.206.143@tcp132 [8/256/0/180] [13950.577996] LNet: Added LNI 192.168.206.143@tcp133 [8/256/0/180] [13951.565121] LNet: Added LNI 192.168.206.143@tcp134 [8/256/0/180] [13952.656912] LNet: Added LNI 192.168.206.143@tcp135 [8/256/0/180] [13960.306165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13960.348893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13961.561138] Lustre: 437687:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.143@tcp: can't add NID 192.168.206.143@tcp130: rc = -75 [13961.659290] LNetError: 437708:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13961.674154] LNetError: Protocol error connecting to 192.168.206.143@tcp115 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13961.684924] LNetError: Skipped 1 previous similar message [13961.782446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13961.820573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13961.824221] LNet: 3 local NIs in recovery (showing 3): 192.168.206.143@tcp126, 192.168.206.143@tcp128, 192.168.206.143@tcp129 [13961.918834] Lustre: lustre-MDT0000: new disk, initializing [13962.043532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13962.058364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13962.721947] LNetError: 437709:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13962.734531] LNetError: 437709:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [13962.745698] LNetError: Protocol error connecting to 192.168.206.143@tcp100 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13962.760152] LNetError: Skipped 5 previous similar messages [13964.901935] LNetError: 435111:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13964.922212] LNetError: 435111:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13964.933669] LNetError: Protocol error connecting to 192.168.206.143@tcp129 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13964.941374] LNetError: Skipped 3 previous similar messages [13966.617718] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [13968.997969] LNetError: 437710:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13969.032686] LNetError: Protocol error connecting to 192.168.206.143@tcp129 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13978.735884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13978.947254] Lustre: 437706:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13979.031436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13979.035236] Lustre: Skipped 1 previous similar message [13979.148161] Lustre: lustre-MDT0001: new disk, initializing [13979.201201] Lustre: 438489:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.143@tcp: can't add NID 192.168.206.143@tcp130: rc = -75 [13979.217382] Lustre: 438489:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13979.346110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13979.418504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13979.433455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13984.313570] Lustre: 438693:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.143@tcp: can't add NID 192.168.206.143@tcp130: rc = -75 [13984.323739] Lustre: 438693:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13984.617448] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [13991.529258] LNetError: 437709:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13991.541273] LNetError: Protocol error connecting to 192.168.206.143@tcp124 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13992.383586] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13997.505087] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13999.719353] LNetError: 437719:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [13999.741699] LNetError: 437719:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13999.760937] LNetError: Protocol error connecting to 192.168.206.143@tcp115 at host 192.168.206.143:988: is it running a compatible version of Lustre? [13999.773122] LNetError: Skipped 4 previous similar messages [14005.868952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14006.382740] Lustre: lustre-OST0000: new disk, initializing [14006.385761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14006.397767] Lustre: 439452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14006.556496] Lustre: 439457:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.143@tcp: can't add NID 192.168.206.143@tcp130: rc = -75 [14006.569185] Lustre: 439457:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14006.633108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14013.991586] Lustre: 439747:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.143@tcp: can't add NID 192.168.206.143@tcp130: rc = -75 [14014.014309] Lustre: 439747:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14014.112029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14014.133171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14014.342776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14014.677227] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14017.136309] LNetError: 435111:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [14017.159836] LNetError: 435111:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14017.187957] LNetError: Protocol error connecting to 192.168.206.143@tcp124 at host 192.168.206.143:988: is it running a compatible version of Lustre? [14017.212172] LNetError: Skipped 11 previous similar messages [14021.216398] LNet: 1 local NIs in recovery (showing 1): 192.168.206.143@tcp129 [14024.954740] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14034.914160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14034.921508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14034.923460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14034.935038] Lustre: Skipped 1 previous similar message [14040.045403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14040.052018] Lustre: Skipped 2 previous similar messages [14040.761897] Lustre: server umount lustre-OST0000 complete [14045.163910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14045.175233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14045.176487] Lustre: Skipped 1 previous similar message [14045.209434] Lustre: Skipped 1 previous similar message [14050.277421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14050.280367] Lustre: Skipped 1 previous similar message [14051.557449] Lustre: server umount lustre-MDT0000 complete [14053.985505] LNetError: 435113:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.143 [14054.002345] LNetError: 435113:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14054.014816] LNetError: Protocol error connecting to 192.168.206.143@tcp100 at host 192.168.206.143:988: is it running a compatible version of Lustre? [14054.027947] LNetError: Skipped 13 previous similar messages [14055.644638] LustreError: 437696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779140481 with bad export cookie 498500462244442112 [14055.648231] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14055.678086] LustreError: 437696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14056.131505] Lustre: server umount lustre-MDT0001 complete [14069.166664] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [14072.537885] Key type lgssc unregistered [14073.006439] LNet: 441240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14073.031594] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14074.080277] LNet: Removed LNI 192.168.206.143@tcp [14074.090061] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14074.106161] LNet: Removed LNI 192.168.206.143@tcp100 [14075.168571] LNet: Removed LNI 192.168.206.143@tcp101 [14075.178944] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14075.192408] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14075.210109] LNet: Removed LNI 192.168.206.143@tcp102 [14076.256278] LNet: Removed LNI 192.168.206.143@tcp103 [14077.280174] LNet: Removed LNI 192.168.206.143@tcp104 [14077.282429] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14077.296870] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14077.307700] LNet: Removed LNI 192.168.206.143@tcp105 [14078.368189] LNet: Removed LNI 192.168.206.143@tcp106 [14079.392451] LNet: Removed LNI 192.168.206.143@tcp107 [14080.416327] LNet: Removed LNI 192.168.206.143@tcp108 [14081.442181] LNet: Removed LNI 192.168.206.143@tcp109 [14081.457355] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14081.476324] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14081.508303] LNet: Removed LNI 192.168.206.143@tcp110 [14082.529073] LNet: Removed LNI 192.168.206.143@tcp111 [14083.552507] LNet: Removed LNI 192.168.206.143@tcp112 [14084.583219] LNet: Removed LNI 192.168.206.143@tcp113 [14085.600218] LNet: Removed LNI 192.168.206.143@tcp114 [14086.626088] LNet: Removed LNI 192.168.206.143@tcp115 [14087.648172] LNet: Removed LNI 192.168.206.143@tcp116 [14088.672414] LNet: Removed LNI 192.168.206.143@tcp117 [14089.697263] LNet: Removed LNI 192.168.206.143@tcp118 [14089.704586] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14089.727426] LNetError: 441240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14089.743570] LNet: Removed LNI 192.168.206.143@tcp119 [14090.785248] LNet: Removed LNI 192.168.206.143@tcp120 [14091.809388] LNet: Removed LNI 192.168.206.143@tcp121 [14092.832215] LNet: Removed LNI 192.168.206.143@tcp122 [14093.856429] LNet: Removed LNI 192.168.206.143@tcp123 [14094.880275] LNet: Removed LNI 192.168.206.143@tcp124 [14095.904196] LNet: Removed LNI 192.168.206.143@tcp125 [14096.928245] LNet: Removed LNI 192.168.206.143@tcp126 [14097.952162] LNet: Removed LNI 192.168.206.143@tcp127 [14098.976650] LNet: Removed LNI 192.168.206.143@tcp128 [14098.978918] LNet: Removed LNI 192.168.206.143@tcp129 [14098.986500] LNet: Removed LNI 192.168.206.143@tcp130 [14098.997242] LNet: Removed LNI 192.168.206.143@tcp131 [14098.999361] LNet: Removed LNI 192.168.206.143@tcp132 [14099.008513] LNet: Removed LNI 192.168.206.143@tcp133 [14099.012831] LNet: Removed LNI 192.168.206.143@tcp134 [14099.025804] LNet: Removed LNI 192.168.206.143@tcp135 [14099.813093] Key type .llcrypt unregistered [14099.815459] Key type ._llcrypt unregistered [14124.660703] Key type ._llcrypt registered [14124.662511] Key type .llcrypt registered [14124.738140] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [14142.140356] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [14143.527500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14143.586651] alg: No test for adler32 (adler32-zlib) [14144.787579] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [14145.099558] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [14146.883050] Key type lgssc registered [14148.206857] Lustre: Echo OBD driver; http://www.lustre.org/ [14193.854942] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:43:37 (1779140617) [14216.411954] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [14219.302414] Key type lgssc unregistered [14219.770606] LNet: 445332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14219.785716] LNetError: 445332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14219.803287] LNet: Removed LNI 192.168.206.143@tcp [14220.597729] Key type .llcrypt unregistered [14220.605185] Key type ._llcrypt unregistered [14239.845314] Key type ._llcrypt registered [14239.849869] Key type .llcrypt registered [14239.957422] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [14241.210768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14241.242083] alg: No test for adler32 (adler32-zlib) [14242.304473] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [14242.583336] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [14244.304418] Key type lgssc registered [14245.270866] Lustre: Echo OBD driver; http://www.lustre.org/ [14255.442502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14255.475269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14256.825634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14256.872857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14256.948661] Lustre: lustre-MDT0000: new disk, initializing [14257.072789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14257.094417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14261.341418] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14271.196929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14271.277985] Lustre: 446582:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14271.331755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14271.338272] Lustre: Skipped 1 previous similar message [14271.432429] Lustre: lustre-MDT0001: new disk, initializing [14271.536982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14271.581936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14271.594866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14275.660618] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14284.770323] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14289.572698] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14299.247896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14299.533693] Lustre: lustre-OST0000: new disk, initializing [14299.537896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14299.548509] Lustre: 448326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14299.641869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14305.368080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14305.376168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14305.525139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14306.345415] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14313.856794] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14318.142577] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing lnet_if_list [14323.875675] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14323.916300] LNet: Added LNI 192.168.206.143@tcp10 [8/256/0/180] [14329.753712] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14329.802314] LNet: Added LNI 192.168.206.143@tcp11 [8/256/0/180] [14334.474237] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14334.516669] LNet: Added LNI 192.168.206.143@tcp12 [8/256/0/180] [14339.787578] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14339.833918] LNet: Added LNI 192.168.206.143@tcp13 [8/256/0/180] [14345.247355] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14345.286414] LNet: Added LNI 192.168.206.143@tcp14 [8/256/0/180] [14350.334763] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14350.360756] LNet: Added LNI 192.168.206.143@tcp15 [8/256/0/180] [14354.924801] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14354.977767] LNet: Added LNI 192.168.206.143@tcp16 [8/256/0/180] [14359.404191] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14359.446599] LNet: Added LNI 192.168.206.143@tcp17 [8/256/0/180] [14364.518511] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14364.589749] LNet: Added LNI 192.168.206.143@tcp18 [8/256/0/180] [14369.603313] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14369.652366] LNet: Added LNI 192.168.206.143@tcp19 [8/256/0/180] [14374.827902] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14374.865633] LNet: Added LNI 192.168.206.143@tcp20 [8/256/0/180] [14380.393743] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14380.438021] LNet: Added LNI 192.168.206.143@tcp21 [8/256/0/180] [14385.684993] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14385.713547] LNet: Added LNI 192.168.206.143@tcp22 [8/256/0/180] [14391.727617] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14391.767275] LNet: Added LNI 192.168.206.143@tcp23 [8/256/0/180] [14398.070441] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14398.107532] LNet: Added LNI 192.168.206.143@tcp24 [8/256/0/180] [14404.242476] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14404.266525] LNet: Added LNI 192.168.206.143@tcp25 [8/256/0/180] [14410.613055] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14410.647303] LNet: Added LNI 192.168.206.143@tcp26 [8/256/0/180] [14417.525509] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14417.565669] LNet: Added LNI 192.168.206.143@tcp27 [8/256/0/180] [14424.724411] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14424.766836] LNet: Added LNI 192.168.206.143@tcp28 [8/256/0/180] [14431.816973] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14431.866691] LNet: Added LNI 192.168.206.143@tcp29 [8/256/0/180] [14438.329719] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14438.361068] LNet: Added LNI 192.168.206.143@tcp30 [8/256/0/180] [14445.368814] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14445.418506] LNet: Added LNI 192.168.206.143@tcp31 [8/256/0/180] [14452.528523] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14452.555514] LNet: Added LNI 192.168.206.143@tcp32 [8/256/0/180] [14459.001272] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14459.029579] LNet: Added LNI 192.168.206.143@tcp33 [8/256/0/180] [14466.289832] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14466.324081] LNet: Added LNI 192.168.206.143@tcp34 [8/256/0/180] [14472.553306] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14472.582742] LNet: Added LNI 192.168.206.143@tcp35 [8/256/0/180] [14479.195391] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14479.224051] LNet: Added LNI 192.168.206.143@tcp36 [8/256/0/180] [14486.358725] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14486.392033] LNet: Added LNI 192.168.206.143@tcp37 [8/256/0/180] [14495.500243] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14495.551663] LNet: Added LNI 192.168.206.143@tcp38 [8/256/0/180] [14503.395088] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14503.429624] LNet: Added LNI 192.168.206.143@tcp39 [8/256/0/180] [14510.790545] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14510.832325] LNet: Added LNI 192.168.206.143@tcp40 [8/256/0/180] [14518.108451] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14518.147537] LNet: Added LNI 192.168.206.143@tcp41 [8/256/0/180] [14525.366602] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14525.410942] LNet: Added LNI 192.168.206.143@tcp42 [8/256/0/180] [14532.876153] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14532.907658] LNet: Added LNI 192.168.206.143@tcp43 [8/256/0/180] [14541.333225] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14541.374745] LNet: Added LNI 192.168.206.143@tcp44 [8/256/0/180] [14549.963604] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14549.995415] LNet: Added LNI 192.168.206.143@tcp45 [8/256/0/180] [14559.012512] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14559.066967] LNet: Added LNI 192.168.206.143@tcp46 [8/256/0/180] [14568.724288] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14568.774107] LNet: Added LNI 192.168.206.143@tcp47 [8/256/0/180] [14579.381162] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14579.479853] LNet: Added LNI 192.168.206.143@tcp48 [8/256/0/180] [14588.355602] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14588.389467] LNet: Added LNI 192.168.206.143@tcp49 [8/256/0/180] [14597.176771] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14597.233105] LNet: Added LNI 192.168.206.143@tcp50 [8/256/0/180] [14618.088127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14618.120065] Lustre: Skipped 1 previous similar message [14618.122124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14618.140857] Lustre: Skipped 2 previous similar messages [14623.203089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14623.212989] Lustre: Skipped 2 previous similar messages [14623.762853] Lustre: server umount lustre-MDT0000 complete [14627.996048] LustreError: 446577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779141053 with bad export cookie 2578593753168757524 [14627.999404] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14628.012604] LustreError: 446577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14628.327290] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14628.350834] Lustre: Skipped 1 previous similar message [14628.565433] Lustre: server umount lustre-MDT0001 complete [14649.878318] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [14658.728992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14672.704879] LustreError: 448319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14672.731820] LustreError: 448319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14672.770372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14673.763725] LustreError: 448320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14676.882874] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14679.014423] LustreError: 448319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14684.141547] LustreError: 448320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14684.161091] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14685.717400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14686.146798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14691.351949] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14693.367941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14697.465294] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14701.575182] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14722.018926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14722.031920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14722.059325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14724.068095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14724.069403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14724.090150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14729.186255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14729.202350] Lustre: Skipped 2 previous similar messages [14736.352145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14736.601564] Lustre: server umount lustre-OST0000 complete [14742.505373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14742.518243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14742.534149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14742.544458] Lustre: Skipped 2 previous similar messages [14746.727597] Lustre: server umount lustre-MDT0000 complete [14749.667047] LustreError: 471735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14749.682981] LustreError: 471735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14750.625759] LustreError: 446576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779141176 with bad export cookie 2578593753168760058 [14750.635525] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14750.639535] LustreError: 446576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14751.005109] Lustre: server umount lustre-MDT0001 complete [14762.295185] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [14765.311255] Key type lgssc unregistered [14765.736494] LNet: 474294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14765.755555] LNetError: 474294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14766.816567] LNet: Removed LNI 192.168.206.143@tcp [14766.820310] LNetError: 474294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14766.826331] LNet: Removed LNI 192.168.206.143@tcp10 [14766.839431] LNet: Removed LNI 192.168.206.143@tcp11 [14766.854569] LNet: Removed LNI 192.168.206.143@tcp12 [14766.858760] LNet: Removed LNI 192.168.206.143@tcp13 [14766.860956] LNet: Removed LNI 192.168.206.143@tcp14 [14766.865683] LNet: Removed LNI 192.168.206.143@tcp15 [14766.871320] LNet: Removed LNI 192.168.206.143@tcp16 [14766.875171] LNet: Removed LNI 192.168.206.143@tcp17 [14766.886993] LNet: Removed LNI 192.168.206.143@tcp18 [14766.895204] LNet: Removed LNI 192.168.206.143@tcp19 [14766.900940] LNet: Removed LNI 192.168.206.143@tcp20 [14766.906251] LNet: Removed LNI 192.168.206.143@tcp21 [14766.910996] LNet: Removed LNI 192.168.206.143@tcp22 [14766.918669] LNet: Removed LNI 192.168.206.143@tcp23 [14766.922763] LNet: Removed LNI 192.168.206.143@tcp24 [14766.926095] LNet: Removed LNI 192.168.206.143@tcp25 [14766.929230] LNet: Removed LNI 192.168.206.143@tcp26 [14766.935352] LNet: Removed LNI 192.168.206.143@tcp27 [14766.942534] LNet: Removed LNI 192.168.206.143@tcp28 [14766.945989] LNet: Removed LNI 192.168.206.143@tcp29 [14766.951235] LNet: Removed LNI 192.168.206.143@tcp30 [14766.954611] LNet: Removed LNI 192.168.206.143@tcp31 [14766.958647] LNet: Removed LNI 192.168.206.143@tcp32 [14766.962786] LNet: Removed LNI 192.168.206.143@tcp33 [14766.966767] LNet: Removed LNI 192.168.206.143@tcp34 [14766.970692] LNet: Removed LNI 192.168.206.143@tcp35 [14766.973737] LNet: Removed LNI 192.168.206.143@tcp36 [14766.977785] LNet: Removed LNI 192.168.206.143@tcp37 [14766.981714] LNet: Removed LNI 192.168.206.143@tcp38 [14766.985641] LNet: Removed LNI 192.168.206.143@tcp39 [14766.989427] LNet: Removed LNI 192.168.206.143@tcp40 [14766.996930] LNet: Removed LNI 192.168.206.143@tcp41 [14767.003812] LNet: Removed LNI 192.168.206.143@tcp42 [14767.011658] LNet: Removed LNI 192.168.206.143@tcp43 [14767.019357] LNet: Removed LNI 192.168.206.143@tcp44 [14767.022784] LNet: Removed LNI 192.168.206.143@tcp45 [14767.026196] LNet: Removed LNI 192.168.206.143@tcp46 [14767.030410] LNet: Removed LNI 192.168.206.143@tcp47 [14767.034566] LNet: Removed LNI 192.168.206.143@tcp48 [14767.037257] LNet: Removed LNI 192.168.206.143@tcp49 [14767.049703] LNet: Removed LNI 192.168.206.143@tcp50 [14767.893843] Key type .llcrypt unregistered [14767.898683] Key type ._llcrypt unregistered [14789.575445] Key type ._llcrypt registered [14789.578223] Key type .llcrypt registered [14789.719548] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [14807.000759] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [14808.292171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14808.358115] alg: No test for adler32 (adler32-zlib) [14809.628369] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [14809.936840] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [14811.641841] Key type lgssc registered [14813.003941] Lustre: Echo OBD driver; http://www.lustre.org/ [14853.445703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14853.487704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14854.857621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14854.941477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14855.139436] Lustre: lustre-MDT0000: new disk, initializing [14855.285166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14855.304294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14860.351275] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14873.415387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14873.574732] Lustre: 477510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14873.656692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14873.665792] Lustre: Skipped 1 previous similar message [14873.780144] Lustre: lustre-MDT0001: new disk, initializing [14873.948296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14874.021391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14874.034782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14878.849796] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14887.134975] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14891.932763] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14901.226705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14901.487343] Lustre: *** cfs_fail_loc=90f, val=0*** [14901.580616] Lustre: lustre-OST0000: new disk, initializing [14901.588983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14901.599604] Lustre: 479302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14901.682907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14908.016292] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [14911.072826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14911.088289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14911.228562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14916.002104] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14926.821797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14926.831613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14926.836209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14926.854689] Lustre: Skipped 1 previous similar message [14930.189541] Lustre: server umount lustre-OST0000 complete [14935.533347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14935.548687] LustreError: Skipped 1 previous similar message [14935.556426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14935.579211] Lustre: Skipped 1 previous similar message [14935.597762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14937.057987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14937.064987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14940.476979] Lustre: server umount lustre-MDT0000 complete [14944.573957] LustreError: 479301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779141370 with bad export cookie 16511263191136847053 [14944.584627] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14944.591462] LustreError: 479301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14944.920854] Lustre: server umount lustre-MDT0001 complete [14956.700344] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [14959.636799] Key type lgssc unregistered [14959.896985] LNet: 481079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14959.903597] LNetError: 481079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14959.930630] LNet: Removed LNI 192.168.206.143@tcp [14960.682787] Key type .llcrypt unregistered [14960.686069] Key type ._llcrypt unregistered [14981.803045] Key type ._llcrypt registered [14981.805795] Key type .llcrypt registered [14981.938169] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [14983.915813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14983.948567] alg: No test for adler32 (adler32-zlib) [14985.266319] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [14985.651799] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [14987.552182] Key type lgssc registered [14990.120188] Lustre: Echo OBD driver; http://www.lustre.org/ [15003.106246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15003.161786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15005.112728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15011.556398] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15014.385650] LustreError: 482390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15014.440673] LustreError: 482390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15019.499099] LustreError: 482391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15022.695602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15023.035536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15027.867944] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15034.019816] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15038.501643] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15045.983150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15046.330063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15053.054111] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15059.324118] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15072.807336] loop0: detected capacity change from 0 to 204800000 [15073.104638] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15073.124872] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15073.161595] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15079.742570] loop0: detected capacity change from 0 to 204800000 [15079.943190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15079.953400] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15079.966054] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15085.398702] loop0: detected capacity change from 0 to 204800000 [15092.191846] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15092.444228] Lustre: fs15246-OST0000: new disk, initializing [15092.457683] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15092.491567] Lustre: 485647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15092.596420] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15099.675980] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15109.968223] loop1: detected capacity change from 0 to 204800000 [15116.278906] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15116.434879] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15116.529791] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15116.640757] Lustre: fs15246-MDT0000: new disk, initializing [15116.775992] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15116.799980] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15116.910882] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15117.993197] Lustre: 486753:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000075f45d0d x1865564984230016/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779141559 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15122.555950] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15123.110231] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15123.124906] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15123.185434] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15147.506663] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15147.540890] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15151.225930] Lustre: server umount fs15246-MDT0000 complete [15152.617848] LustreError: 482385:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15157.741140] LustreError: 487951:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15160.706884] Lustre: server umount fs15246-OST0000 complete [15172.579416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15172.588455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15172.606892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15178.053927] Lustre: server umount lustre-OST0000 complete [15181.793259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15181.798182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15181.804581] Lustre: Skipped 1 previous similar message [15181.810335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15181.813359] Lustre: Skipped 2 previous similar messages [15188.290503] Lustre: server umount lustre-MDT0000 complete [15188.478342] LustreError: 482391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15188.500384] LustreError: 482391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15192.575551] LustreError: 482371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779141618 with bad export cookie 4181887138589957670 [15192.577934] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15192.594282] LustreError: 482371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15193.053695] Lustre: server umount lustre-MDT0001 complete [15206.006396] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [15208.921710] Key type lgssc unregistered [15209.231901] LNet: 489815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15209.239187] LNetError: 489815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15210.287183] LNet: Removed LNI 192.168.206.143@tcp [15211.155912] Key type .llcrypt unregistered [15211.159779] Key type ._llcrypt unregistered [15231.537287] Key type ._llcrypt registered [15231.553313] Key type .llcrypt registered [15231.697773] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [15233.125826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15233.173662] alg: No test for adler32 (adler32-zlib) [15234.267894] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [15234.486272] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [15236.184533] Key type lgssc registered [15237.588732] Lustre: Echo OBD driver; http://www.lustre.org/ [15282.571716] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [15292.767237] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [15334.219324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15334.274210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15335.527498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15335.556959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15335.617899] Lustre: lustre-MDT0000: new disk, initializing [15335.724327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15335.749404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15340.523803] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15352.815070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15352.953309] Lustre: 494304:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15352.998996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15353.003631] Lustre: Skipped 1 previous similar message [15353.083916] Lustre: lustre-MDT0001: new disk, initializing [15353.192455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15353.239649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15353.259847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15357.416181] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15365.243156] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15369.439485] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15377.122670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15377.430180] Lustre: lustre-OST0000: new disk, initializing [15377.433804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15377.441417] Lustre: 496050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15377.542090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15383.080079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15383.101278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15383.272511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15383.964430] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [15392.327779] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15397.185921] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15397.493240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15402.606795] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15402.898535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15409.121250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15409.127460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15409.146899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15409.153269] Lustre: Skipped 1 previous similar message [15414.259124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15414.273387] Lustre: Skipped 1 previous similar message [15419.365284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15419.366555] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [15419.380359] Lustre: Skipped 1 previous similar message [15419.653310] Lustre: server umount lustre-OST0000 complete [15424.484674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15424.487841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15424.507670] Lustre: Skipped 2 previous similar messages [15424.529101] Lustre: Skipped 1 previous similar message [15429.605514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15429.614025] Lustre: Skipped 1 previous similar message [15430.048605] Lustre: server umount lustre-MDT0000 complete [15435.282083] LustreError: 494299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779141861 with bad export cookie 16561113183927656003 [15435.300903] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15435.951665] Lustre: server umount lustre-MDT0001 complete [15466.974094] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [15479.868095] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [15516.221562] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15249 sec ========= 18:05:39 (1779141939) [15517.990864] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:05:41 (1779141941) === [15522.028612] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:05:45 (1779141945) === [15560.161745] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [15563.176635] Key type lgssc unregistered [15563.502666] LNet: 502509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15563.515710] LNetError: 502509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15563.530839] LNet: Removed LNI 192.168.206.143@tcp [15564.558640] Key type .llcrypt unregistered [15564.560977] Key type ._llcrypt unregistered